Artificial intelligence and/or virtual reality for activity optimization/personalization

ABSTRACT

Optimizing and/or personalizing activities to a user through artificial intelligence and/or virtual reality.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of U.S. ProvisionalApplication No. 62/552,096, filed Aug. 30, 2017, and U.S. ProvisionalApplication No. 62/552,091, filed Aug. 30, 2017, the entireties of whichare hereby incorporated by reference.

FIELD OF THE DISCLOSURE

The field of the disclosure relates generally to artificial intelligenceand virtual/augmented reality, and more specifically, to methods andsystems for optimizing/personalizing user activities through artificialintelligence and/or virtual reality.

BACKGROUND

Artificial intelligence (AI) is slowly being incorporated into themedical field. AI systems and techniques can be used to improve healthservices—both at the physician and patient levels, used to improve theaccuracy of medical diagnosis, manage treatments, provide real timemonitoring of patients, and integrate the different health providers andhealth services together—all while decreasing the costs of medicalservices. However, some previous attempts to use artificial intelligencein medicine have failed. For example, IBM's Watson attempted to useartificial intelligence techniques for oncology therapy. Watson failedto help oncologic treatments because Watson was not a linear artificialintelligence dictated purely by logic and was unable to analyzevariances in biologic functions at a variety of different levels.

BRIEF DESCRIPTION

In an aspect, a system includes a monitor device and an artificialintelligence (AI) system. The monitor device is configured to monitorone or more physical properties of a user. The AI system is configuredto receive and analyze the monitored physical properties to generate oneor more activity parameters optimized or personalized to the user. TheAI system is configured to implement one or more artificial intelligencetechniques such as predictive learning, machine learning, automatedplanning and scheduling, machine perception, computer vision, affectivecomputing to generate one or more activity parameters optimized orpersonalized to a user.

In another aspect, a system includes a goggle device and a controller.The goggle device is configured to provide one or more images to a userof the system and perform at least one vision test on the user. Thecontroller is configured to execute an algorithm for tracking at leastone vision-related impairment of the user based on the vision testand/or enhancing the vision of the user based on the vision test.

In another aspect, a method of diagnosing diseases and assessing healthis performed by retinal imaging or scanning. The pupil is dilated by,for example, dark glasses, and then the retina is imaged. In anembodiment, the image of the retina is evaluated by a computing device,a person, or both to glean information relating to not only health, butalso emotional reactions, physiological reactions, etc.

In another aspect, optical imaging, especially of the retina, is used toobtain real-time feedback data, which can be analyzed by a computer, aperson, or both to determine emotional response, pain, etc. Thisfeedback data can be fed into a VR/AR program or otherwise used todetermine a subject's emotional and/or physiological response to certainstimuli.

In still another aspect, optical imaging or scanning is used forcontinuous health monitoring. For example, continuous or regular imagingof the eye is used to track blood pressure. Reactions to a stimulus,such as exercise for example, could help doctors and could be utilizedby a computer to automatically check for signs of disease and/or poorhealth in various areas. In an embodiment, findings are integrated withother systems, such as those used to collect medical data for example,to provide more accurate and/or comprehensive findings.

In yet another aspect, a retina is evaluated to allow a computer to makeadjustments based on real-time user feedback. For example, if the personis playing a VR/AR game, a processor can use instantaneous feedback fromthe user to adjust difficulty, pace, etc.

In another aspect, alternate methods of measuring blood pressure andother health statistics are used for continual monitoring. In anexemplary and non-limiting embodiment, a wrist-wearable monitor or anearpiece monitor with a Doppler ultrasound imaging system is adapted toestimate blood pressure. In a preferred embodiment, continual retinalimaging and evaluation and health monitoring devices work in combinationwith a device to continuously measure blood pressure. In such anembodiment, a processor will preferably have access to any data gatheredby retinal imaging and/or other health monitoring devices.

The features, functions, and advantages that have been discussed can beachieved independently in various embodiments or may be combined in yetother embodiments, further details of which can be seen with referenceto the following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary system and data flowaccording to an embodiment.

FIG. 2 is a block diagram of an exemplary waking/alerting algorithmaccording to an embodiment.

FIG. 3 is a block diagram of an exemplary automated blood pressuremeasurement algorithm according to an embodiment.

FIG. 4 is a block diagram of an exemplary blood draw algorithm andsystem according to an embodiment.

FIG. 5 illustrates an exemplary architecture of a computing deviceconfigured to provide aspects of the systems and processes describedherein via a software environment.

FIGS. 6-12 illustrate an exemplary virtual reality system according toan embodiment.

FIG. 13 is an image of an Amsler Grid showing a large scotomaencroaching on the central fixation.

FIG. 14 is an image of the peripheral vision of the human eye.

FIG. 15 is an image of scotoma affecting vision.

FIG. 16 is a modified image to correct for scotoma according to anembodiment.

FIG. 17 is an image of a histoplasmosis scar.

FIG. 18 is an image of vision distortion caused by the scar of FIG. 17 .

FIG. 19 is an image of adjusted vision distortion with a filteraccording to an embodiment.

FIG. 20 is a flowchart of an exemplary algorithm for filtering cameradata according to an embodiment.

FIG. 21 is an example of conventional cellular phone apps that providevision testing and/or Amsler grid progression testing.

FIG. 22 is an image of an exemplary goggle device (front and rear view),display, and mouse/pad according to an embodiment.

FIG. 23 is a block diagram of an exemplary goggle device controllermethod and system according to an embodiment.

FIG. 24 is a schematic illustration of a retinal evaluation systemaccording to an embodiment.

FIG. 25A is a schematic block diagram of an exemplary embodiment of theretinal evaluation system of FIG. 24 .

FIG. 25B is a schematic block diagram of a retinal scanning systemincluded in an alternative exemplary embodiment of the retinalevaluation system of FIG. 24 .

FIG. 26 is a schematic illustration of a stimulus response measurementsystem according to an embodiment.

FIG. 27A is a schematic block diagram of the stimulus responsemeasurement system of FIG. 26 .

FIG. 27B is a schematic block diagram of an image creation systemincluded in the embodiment of FIGS. 26 and 27A.

FIG. 28 is a schematic illustration of an earpiece and computer forreal-time measurement of blood pressure according to an embodiment.

FIG. 29 is a schematic block diagram of the earpiece and computer ofFIG. 28 .

FIG. 30A is a schematic illustration of a wearable blood pressuremonitor with a traditional cuff according to an embodiment.

FIG. 30B is a schematic block diagram of a wearable blood pressuremonitor according to an embodiment.

FIG. 31 is a schematic illustration of a wrist-wearable embodiment ofthe wearable blood pressure monitor of FIG. 30B.

FIG. 32 is a flow chart of evaluating exercise through continuousmeasurements according to an embodiment.

FIG. 33 is a flow chart of a method of performing research studies usingreal-time measurements of the subjects of the study to determinereactions according to an embodiment.

FIG. 34 is a flow chart of a method of using real-time measurements of acontent viewer to recommend content and select relevant ads according toan embodiment.

FIG. 35 is a flow chart of a method of adjusting a simulation based onreal-time measurements of the user according to an embodiment.

FIG. 36 is a flow chart of a method of continuously monitoring health bytaking real-time measurements of the user according to an embodiment.

Corresponding reference characters indicate corresponding partsthroughout the drawings.

DETAILED DESCRIPTION

The systems and methods described herein, in an embodiment, enable theoptimization and/or personalization of health-related tasks throughartificial intelligence (AI). Aspects described herein also enableoptimization and/or personalization in microclimate, robotics,management information systems, and the like.

FIG. 1 is a block diagram illustrating an exemplary system 100 foroptimizing and/or personalizing health-related tasks, for example. Thesystem 100 includes one or more patient monitor sensors 102, an AIsystem 104, a dietary database 106, a pharmacy-controlled medicationdelivery subsystem 108, an electronic medical records database 110, aglobal expert system 112, a patient-controlled medication deliverysubsystem 114, a call button 116, a smart alert system 118, one or morehealthcare provider devices 120, and one or more patient devices 122. Inan embodiment, system 100 enables automated planning and scheduling(e.g., AI planning) of strategies or action sequences for execution byhealthcare providers and/or patients such that delivery of healthcareservices is optimized (e.g., optimal for a healthcare provider and/orgroup of healthcare providers, optimal for the patient care and/orsatisfaction, etc.) and/or personalized (e.g., personalized toneeds/requirements of a healthcare provider and/or group of healthcareproviders, personalized to needs/requirements of the patient, etc.).

The patient monitor sensors 102, the dietary database 106, thepharmacy-controlled medication delivery subsystem 108, the electronicmedical records database 110, the global expert system 112, and thepatient-controlled medication delivery subsystem 114 are electricallyand/or communicatively coupled to the AI system 104. Additionally oralternatively, healthcare provider devices 120 and/or patient devices122 are electrically and/or communicatively coupled to AI system 104.The patient monitor sensors 102, the AI system 104, and the call button116 are electrically and/or communicatively coupled to the smart alertsystem 118. The smart alert system 118 is electrically and/orcommunicatively coupled to the healthcare provider devices 120 and thepatient devices 122. In an exemplary and non-limiting embodiment, theelectrical and/or communicative couplings described herein are achievedvia one or more communications networks capable of facilitating theexchange of data among various components of AI system 100. For example,the one or more communications networks may include a wide area network(WAN) that is connectable to other telecommunications networks,including other WANs or portions of the Internet or an intranet,including local area networks (LANs) and/or personal area networks(PANs). The one or more communications networks may be anytelecommunications network that facilitates the exchange of data, suchas those that operate according to the IEEE 802.3 (e.g., Ethernet), theIEEE 802.11 (e.g., Wi-Fi™) and/or the IEEE 802.15 (e.g., Bluetooth®)protocols, for example. In another embodiment, the one or morecommunications networks are any medium that allows data to be physicallytransferred through serial or parallel communication channels (e.g.,copper wire, optical fiber, computer bus, wireless communicationchannel, etc.).

The patient monitor sensors 102 are configured to sense physicalproperties associated with the patient. The patient monitor sensors 102can be generally any type of biometric sensor that generates biometricdata and may be positioned outside or inside the body of a patient.Exemplary sensors include, but are not limited to, contactless bedsensors such as the Murata SCA11H, activity trackers (e.g.,wireless-enabled wearable devices available from Fitbit, Inc., etc.),smartwatches (e.g., the Apple® Watch available from Apple, Inc., etc.),smartphone computing devices, tablet computing devices, smart rings(e.g., MOTA® DOI SmartRing available from Mota Group, Inc., Tokenavailable from Tokenize Inc., etc.), smart glasses, smart contactlenses, video cameras, implants, retinal scanners, flexible sensors,surgical implants, medical implants, voice/sound input (e.g.,microphones), accelerometers, goniometers, and like commercial or customtracking devices with the ability to record and/or transmit patientmetrics (e.g., distance walked or ran, calorie consumption, heartbeat,quality of sleep, movements, sleep patterns, blood pressure, pulse,sweat, skin resistance, etc.). Exemplary sensors further include, butare not limited to, existing sensors used in hospital monitoringsystems, such as hospital records, pulse oximeters, retinal changes,implantable defibrillators, temperature, thermal gradients, changes indiet or food patterns (e.g., from dieticians), medication (e.g., fromthe pharmacy), test results from lab service (e.g., blood work andurinalysis) and the like. Additional exemplary sensors include, but arenot limited to, devices configured to collect data relative tomedications and/or exercise, such as motion patterns, eye movement, bodytemperature, core vs. peripheral breathing, shaking and/or tremors,heart rate, cardiac rhythms and/or arrhythmias, blood pressure, pulse,oximeters, respiration rate, diaphragm excursion, stride length, sleeppatterns (e.g., EMGs, EEGs, etc.), oxygenation (e.g., pulse odometers),hair follicle movement and/or position change, lactic acidosis inmuscles locally and/or systemically, sweat, thermal changes to skinand/or deep tissue, salinity or particles, blood flow, vasoconstriction,vasodilation, foot orthotic sensors on stride length, frequency, load,where load is applied, timing between steps, asymmetry in gait cadenceand/or timing cadence, arm movement, pupillary and/or retinal response,retinal vascular changes, cheek movement (e.g., for retained airresistance, etc.), thermal gradients between one body part and another(e.g., quadriceps and chest or neck, etc.), blood flow between differentbody parts (e.g., neck and foot, etc.) such as measurements from laserflow sensors, ultrasonic sensors, acoustic sensors, electromagneticfield sensors, tension sensors, compression sensors, magnetic resonanceimaging (MRI), positron emission tomography (PET), and the like. Furtherexemplary sensors include, but are not limited to, one or more aspectsof a virtual reality system as further described herein. In anembodiment, a single patient monitor sensor 102 may provide a pluralityof data points. In an embodiment, patient monitor sensors 102 transmitand/or provide data to other aspects of system 100 via wireless, radiofrequency (RF), optical, and the like communications means. Further, ifthe sensor is implanted, the sensor can generate electricity byelectromagnets, motion analysis and/or thermal changes. Moreover, thesensors are not limited to use with a patient and can be used incellular testing, animal testing, and bacterial testing.

Accordingly, aspects of system 100, through the one or more patientmonitors 102, enable patient properties such as biometric data, cellulardata, biologic data, and non-biologic data to be collected and analyzed.This data can then be utilized by the AI system 104 to optimize and/orpersonalize health-related tasks, as described herein. The datacollected can relate to any patient property such as body functions,organ function, cellular functions, and metabolic functions, forexample. Moreover, aspects of system 100 are not limited to people andcan be used for any biologic function. For example, aspects of system100 can be used to analyze animal biologic functions and/ormicrobiologic functions. In all embodiments the capture of informationcould be done via wireless communication, or wired communication. Theinformation could be uploaded to and stored in a central repository orprocessed on site.

The AI system 104 is configured to implement one or more artificialintelligence techniques (e.g., predictive learning, machine learning,automated planning and scheduling, machine perception, computer vision,affective computing, etc.) that optimize and/or personalize one or moreaspects of monitoring, diagnosis, treatment, and prevention of disease,illness, injury, physical and/or mental impairments of the patient. Inan embodiment, AI system 104 comprises processor-executable instructionsembodied on a storage memory device of a computing device to providepredictive learning techniques via a software environment. For example,AI system 104 may be provided as processor-executable instructions thatcomprise a procedure, a function, a routine, a method, and/or asubprogram utilized independently or in conjunction with additionalaspects of system 100 according to an exemplary embodiment of thedisclosure. Additional details regarding AI system 104 are providedherein.

The dietary database 106 is configured to store an organized collectionof data representing one or more of a dietary history (e.g., food and/ornutrient consumption levels, etc.) of the patient, dietary preferencesof the patient, food and/or nutrient consumption levels of populationsin a given geographic area (e.g., worldwide, in a geographic locality ofthe patient, etc.), food composition (e.g., USDA National NutrientDatabase for Standard Reference, USDA Branded Food Products Database,etc.), dietary supplement labels (e.g., Dietary Supplement LabelDatabase from the National Institutes of Health), and the like.

The pharmacy-controlled medication delivery subsystem 108 is configuredto allow a pharmacy actor (e.g., pharmacist, pharmacy staff member,pharmacy automated system, etc.) to administer medication to thepatient. In one aspect, the system 100 (e.g., AI system 104) sends datato the pharmacy actor via the pharmacy-controlled medication deliverysubsystem regarding the medication. Such data can include informationrelated to the medication's dosage, type, and administration forexample. In addition, aspects of AI system 104 can be used with systemsand methods of pharmaceutical delivery, such as those described in U.S.Pat. No. 9,750,612, the entire disclosure of which is herebyincorporated by reference.

The electronic medical records database 110 is configured to store anorganized collection of data representing one or more of demographics,medical history, medication history, allergies, immunization status,laboratory test results, radiology images, vital signs, personalstatistics (e.g., age, weight, etc.), billing information, and the likefor the patient and/or an entire population.

The global expert system 112 is configured to emulate decision-makingabilities of one or more human experts regarding one or more aspects ofmonitoring, diagnosis, treatment, and prevention of disease, illness,injury, physical and/or mental impairments of the patient. In anembodiment, global expert system 112 includes a knowledge base of factsand/or rules (e.g., global rule set) for each patient and an inferenceengine that applies the rules to known facts to deduce new facts,explain situations, and the like. In an embodiment, global expert system112 comprises processor-executable instructions embodied on a storagememory device of a computing device to provide predictive learningtechniques via a software environment. For example, global expert system112 may be provided as processor-executable instructions that comprise aprocedure, a function, a routine, a method, and/or a subprogram utilizedindependently or in conjunction with additional aspects of system 100according to an exemplary embodiment of the disclosure. Additionaldetails regarding global expert system 112 are provided herein.

The patient-controlled medication delivery subsystem 114 is configuredto allow the patient to administer his or her own medication. Exemplaryroutes of administration include, but are not limited to, oral,intravenous, epidural, inhaled, nasal, transcutaneous, and the like.Exemplary patient-controlled medication delivery systems include, but isnot limited to, patient-controlled analgesia (PCA), an intravenous (IV)drip system, and the like.

The call button 116 is configured to enable the patient to alert thehealthcare provider (e.g., doctor, nurse, staff member, etc.) of a needfor aid.

The smart alert system 118 is configured to monitor and record physicalproperties associated with the patient (e.g., recent food, movement,sleep pattern, blood pressure, pulse, sweat, skin resistance, etc.)during a time period leading up to an activation of call button 116 bythe patient, compile the monitored and recorded properties into anadaptive system, monitor the physical properties during a future timeperiod, and proactively alert healthcare providers (e.g., via healthcareprovider devices 120) when a similar set of property conditions are met.In this manner, smart alert system 118 is configured to alert healthcareproviders, via healthcare provider devices 120, before the patientpresses call button 116, for example. In an embodiment, smart alertsystem 118 comprises processor-executable instructions embodied on astorage memory device of a computing device to provide predictivelearning techniques via a software environment. For example, smart alertsystem 118 may be provided as processor-executable instructions thatcomprise a procedure, a function, a routine, a method, and/or asubprogram utilized independently or in conjunction with additionalaspects of system 100 according to an exemplary embodiment of thedisclosure. Additional details regarding smart alert system 118 areprovided herein.

The healthcare provider devices 120 are configured to provide access toAI system 104 and/or smart alert system 118 and/or provide alerts fromsmart alert system 118 to the healthcare providers. In an aspect,healthcare provider devices 120 are computing devices including, but notlimited to, smartphone computing devices, smartwatches, tablet computingdevices, desktop computing devices, and the like. Additionally oralternatively, healthcare provider devices 120 may include pagers, alarmclocks, buzzers, lights, printed notifications, and the like.

The patient devices 122 are configured to provide alerts from smartalert system 118 to the patient and/or provide access to AI system 104by the patient. In an aspect, patient devices 122 are computing devicesincluding, but not limited to, smartphone computing devices, activitymonitoring devices, smartwatches, tablet computing devices, desktopcomputing devices, telpad computing devices (e.g., HC7-M Telpadavailable from PLDT Inc., etc.), and the like.

In an embodiment, medical devices are electrically and/orcommunicatively coupled to the AI system 104 and are configured toprovide a medical treatment to a patient. For example, bone stimulators,neuro stimulators, and/or pain stimulators can be connected with the AIsystem 104 and controlled/operated by the AI system to deliveryoptimized and/or personalized patient treatment. In an embodiment, themedical device may be a robotic medical device such as those disclosedby U.S. Pat. No. 9,192,395, which is hereby incorporated by reference inits entirety. For example, aspects of system 100 (e.g., AI system 104)can direct a robotic medical device to deliver blood flow orpharmaceuticals to a specific locations through minimally invasiveapproaches, such as by magnetic guidance. In an embodiment, the medicaldevice may be an endotracheal tube such as those disclosed by U.S. Pat.Nos. 6,820,614 and 7,320,319, both of which are hereby incorporated byreference in their entirety.

In an embodiment, aspects of system 100 enable data for a specificpatient to be compared relative to data (e.g., trends, etc.) for a groupand/or subgroup of patients. Exemplary subgroups include, but are notlimited to, age, gender, race, disease type, multiple disease types(e.g., ASA classification, etc.), and the like. For example, a 60 yearold patient with diabetes and hypertension differs from an 80 year oldpatient with no disease-specific markers. Aspects of system 100 enablecreating data trends for an individual, a subgroup (e.g., defined byhealthcare provider to share and/or compare data, etc.) and a generalgroup (e.g., age, sex, gender, country, location, etc.). For example,aspects of system 100 enable comparisons and identifications ofvariances on an individual basis, group basis, daily basis, nocturnalbasis, day/night basis, based on when people eat and/or exercise and/orwhen people are exposed to different environmental conditions, such assunlight. Moreover, this is just not limited to patient comparisons butcan also include cellular functions and/or bacterial functions such asto optimize growth and/or inhibition.

In an embodiment, data collected by patient monitor sensors 102 isencrypted and/or is covered by regulatory (e.g., HIPPA, etc.)requirements. The data may be associated with the patient or the datamay be anonymous and/or encrypted. Such data may include, but is notlimited to, age, weight, gender, biometrics (e.g., macro, micro,cellular and/or mitochondrial), videos, and/or financials. A patient maychoose to temporarily (e.g., during a hospital stay) and/or for a longterm (e.g., at home) share data for use by aspects of system 100 or thedata can be shared automatically with the system. For example, patientdata may be collated to optimized medical treatments, workout regimensand/or timing, generic vs. specific drugs, neutraceutocals vs. over thecounter drugs vs. no medication vs. workout time, and the like. Inanother embodiment, aspects of system 100 (e.g., AI system 104) utilizedata collected by patient monitor sensors 102 to determine when aworkout is most effective for a patient based on characteristicspersonal to the patient and/or a group to which the patient belongsand/or provides a best response for energy, endurance, and the like. Inanother embodiment, aspects of system 100 (e.g., AI system 104) utilizedata collected by patient monitor sensors 102 to determine when is thebest time for a patient to receive medication (i.e., not just if to takeand dosage). In another embodiment, aspects of system 100 (e.g., AIsystem 104) utilize data collected by patient monitor sensors 102 todetermine effects of food and/or physical activities on medicationdelivery. In another embodiment, aspects of system 100 (e.g., AI system104) utilize data collected by patient monitor sensors 102 to determinewhether a patient should workout and what is the best time to workoutrelative to medications and/or treatments. In aspect, theseconsiderations are important for patients exhibiting multiple diseases,such as cancer and hypertension, diabetes and cardiovascular disease,and the like. In another embodiment, aspects of system 100 (e.g., AIsystem 104) predicts how patterns change over time (e.g., hourly, daily,monthly, etc.) for an individual and/or groups and optimizes efforts forschools, employers, families, churches, other social groups, and thelike. The AI system 104 performs these determinations to optimizehealthcare delivery for an individual patient instead of for healthcareprovider staffing concerns, in an embodiment.

In another embodiment, aspects of system 100 (e.g., AI system 104)utilize data collected by patient monitor sensors 102 to determine anoptimal and/or sub-optimal time for the user (e.g., patient) to study,eat, take medications, sleep, read for comprehension, concentrate, work,rest, socialize, call, text message, diet, eat, what to eat, and thelike. For example, these determinations may be made on datasub-classified based on data points and may change as more data isobtained. In an embodiment, the user (e.g., patient) can activelycontrol and turn on/off as desired.

In another embodiment, aspects of system 100 (e.g., AI system 104)determines how user actions can be modified by diurnal patterns and howto optimize environment, food, medications, local events, and the likeand to predict and/or optimize body function and/or activity. In anotherembodiment, aspects of system 100 (e.g., AI system 104) determine whenis the best time for a surgery or procedure, when to take medications,when to eat food, and the like. In an embodiment, a patient verbalizesdiscomfort (e.g., “I feel sick,” “I have a headache,” etc.) and aspectsof system 100 (e.g., AI system 104) modifies recommendations on when tostudy, read, exercise, take medications, dosage levels, level ofactivity (e.g., how strenuous), and the like. In an embodiment, aspectsof system 100 (e.g., AI system 104) communicate to an employer and/orhealthcare provider how much activity, stress, medications, and the likeis appropriate for an individual/patient. In an embodiment, aspects ofsystem 100 (e.g., AI system 104) give direct feedback to theusers/patients themselves on when, where, and how to complete variousactivities to obtain an optimal effect. In an embodiment, a user/patientcan obtain an image of himself (e.g., a “selfie”) to see facialmovements or activity to determine health-related parameters and/or howactive to be. In another embodiment, aspects of system 100 (e.g., AIsystem 104) utilize information from a reference (e.g., the Old Farmer'sAlmanac, horoscopes, etc.) in the intelligence mix to determine trends,such as diurnal (e.g., best time during day), and the like.

In another embodiment, patient monitor sensors 102 modify midstream soif the patient slept poorly, is under stress, is slower responding toquestions, and the like, aspects of system 100 (e.g., AI system 104)change the patient's activity pattern for that day but not subsequentdays. In this manner, aspects of the disclosure are not just comparingto a group but also with an individual's variation patterns note andmodified on a daily, hourly, and the like basis. For example, if theindividual is hung over he or she will be slower and won't perform aswell during that day. The same concept applies in a hospital setting,school setting, and the like. For example, knowing status of employees(e.g., hung over, sleepy, etc.) affects how the individual is treatedand the employer can staff a shift based on their abilities, problems,and the like.

In another embodiment, aspects of system 100 (e.g., AI system 104)determine if a patient needs a pain medication and if/when they needanxiolytics, anti-inflammatories, or just someone to talk to and/ormusic to pacify. For example, aspects of system 100 (e.g., patientmonitor sensors 102 and/or AI system 104) determine these patterns byeye movement, temperature, sweating, core vs. peripheral movement,sweating palms vs. general sweating, heart rate changes, rate ofbreathing, how deep breathing is, shaking, tremors, tone of voice andthe like. These “tells” (e.g., like in poker) may vary between patientsbut learning their response outside a hospital setting helps inside thehospital setting and/or after surgery and the like. In an embodiment,knowledge of these “tells” by aspects of system 100 also help healthcareproviders (e.g., nurses) respond.

In another embodiment, aspects of system 100 (e.g., AI system 104) canbe used to regulate or control medical devices were a treatment isvaried based on body motion, activity, diet, nutrition, sunlight, and/orenvironmental conditions. Such medical devices may include, for example,neuromuscular stimulators, pain stimulators and/or pacemakers thatdeliver an electrical flow (broadly, treatment) to the patient. Forexample, internal pacemakers simply try to regulate the heart rate to aknown condition using electrical flow. However, pacemakers, generally,are set to regulate the heart rate of a patient to a set rate to treat aheart condition (e.g., atrial-fibrillation or ventricular fibrillationor when the heart as asystolically or has multiple heartbeats in ashorter period of time). Aspects of system 100 can monitor the patientand vary or adjust the heat rate the pacemaker regulates the heart ofthe patient at. For example, AI system 104 can identify when a patientis under a high degree of stress, such as by analyzing data from apatient monitor sensor 102, and control the pacemaker to adjust or alterthe heart rate based on the amount of stress. A change in a patient'sstress level may be due to a fear, apprehension, or exercise. Moreover,AI system 104 may change the heart rate for other conditions such aswhen a patient is eating, moving, or resting. Accordingly, instead of aconstant heart rate set by the pacemaker, the AI system 104 can regulatethe heart rate imposed by the pacemaker based on the needs of thepatient.

In another embodiment, aspects of system 100 (e.g., AI system 104) arenot just limited to patients and can be used in other areas such as foranimals, living cells, bacteria, cell growth, cell culture, tissueculture, and other aspects of microbiology. In addition, the aspectssystem 100 can be used for cellular growth, cellular mechanicsmitochondrial mechanics, bacterial growth, and bacterial functions. Forexample, aspects of system 100 can be used for cellular growth in 3Dprinting applications.

In accordance with one or more embodiments:

-   -   Aspects of system 100 (e.g., patient monitor sensors 102 and/or        AI system 104) monitor physical properties of the patient, such        as patient movement and sleep patterns, and shift drug delivery,        blood pressure measurements, and blood draws, food delivery, and        the like up/back to a predetermined amount of time so they are        performed at an optimal time in the patient's sleep schedule.    -   Aspects of system 100 (e.g., AI system 104) combine sleep        patterns for multiple patients to create an optimal room order        for a phlebotomist, dietician, nurse, food deliverer, and the        like to follow to minimize patient disturbances.    -   When a patient is checked in, a local copy of the global rules        set (e.g., global expert system 112) is created. Aspects of        system 100 (e.g., AI system 104) integrate patient-specific        information in the patient's rule set. Aspects of system 100        (e.g., AI system 104) add adaptive rules based on patient        behaviors.    -   When the patient presses the bed-side call button 116 the signal        is sent to a nurse either via a traditional call system or        through smart alert system 118. Aspects of system 100 (e.g., AI        system 104, smart alert system 118, etc.) would then record the        conditions when the button was pressed (e.g., recent food,        movement, sleep patterns, BP, pulse, sweat, etc.). Aspects of        system 100 (e.g., AI system 104, smart alert system 118, etc.)        uses this information to compile an adaptive system that could        set alerts before the patient presses the call button 116 in the        future. For example, when a similar set of conditions are met or        the conditions are within a threshold the alert would be set. In        one or more embodiments, after the hospital staff is notified,        there is a feedback mechanism for the patient and/or the        hospital staff that would indicate if the alert was a false        alert. Reduced call button presses could indicate that the AI        generated rules are having a positive result. Aspects of system        100 (e.g., AI system 104, smart alert system 118, etc.) would        then use this scoring system to intelligently modify the created        rule set. Other known AI weighting algorithms could be used. The        AI System 104 could also ensure the nurse responds to the call        button 116.    -   Aspects of system 100 include safety measures to ensure that the        alerts generated by AI system 104 and/or smart alert system 118        cannot bypass the global/expert system rules in global expert        system 112.    -   Rules generated by AI system 104 and/or smart alert system 118        are compiled and integrated into global expert system 112.    -   Aspects of system 100 (e.g., AI system 104) monitor the food        and/or water intake patterns of the patient so that changes in        diet, portion size, and the like can be monitored.    -   Aspects of system 100 (e.g., smart alert system 118) queue the        patients information by severity of alert (e.g., higher priority        given to more critical cases). For example, a patient with        critical blood pressure levels would get a higher rating than a        patient who had all normal readings.    -   A hospital medical records system is integrated into system 100        so that all information in a patient's records could be used as        an input to system 100.

A goal of AI is the creation of an intelligent computer system. Theseintelligent systems can be used to optimize systems and methods forhealthcare delivery to provide better care and increase patientsatisfaction. At a high level, AI has been broken into strong AI, whichbelieves machines can be sentient, and weak AI, which does not. Althoughembodiments described herein focus on weak AI, they can also beimplemented with a strong AI system in accordance with one or moreaspects of the disclosure. While the embodiments disclosed herein arerelated to healthcare, it is understood that aspects of system 100 mayalso apply to non-medical applications, such as but not limited toindustrial systems, commercial systems, automotive systems, aerospacesystem and/or entertainment systems.

Data analysis by the AI system 104 can include pure algorithms orindividual or panels that review and comment at specific data analysispoints (“opinion” data). The “opinion” data could be included forfurther analysis or bifurcated into a column with and without expert(e.g., humanistic) data analysis and evaluate conclusions. Humananalysis could be individual specialist or pooled group specialists ordifferent specialists like oncologist then a statistician then economistthen ethics expert. Each can add analysis at certain critical points,and then reanalyze the data for conclusions. The AI system 104 may thenanalyze the human conclusion and compare them to its own. This adds abiological factor to analysis and not pure analysis from data.

In all embodiments there may be an advantage to combining known types ofAI such as expert systems, genetic algorithms, deep learning, andconvolutional neural networks (CNN) to implement a unique approach tothe system. Convolutional neural networks and deep learning can be veryuseful in image recognition, such as recognizing a cat. There are casessuch as robotic surgery, medical diagnosis, or reviewing medicaljournals which may not lend itself to traditional AI methods. Forexample, when using peer reviewed journals to assist in diagnosing amedical condition it may be necessary to perform an interim analysis ofthe data to ensure that all the conditions and symptoms of the patientare being considered or articles which not applicable are beingexcluded. Because interim analysis of traditional CNN is not somethingthat can be easily done due to the encoding of the data. Because of thisit could be preferable to break the CNN into multiple CNN with an expertsystem or evaluation by experts at each stage. The interim could be donecould be done by a single user, or a system could be setup where theresults are done by peer review where multiple users review. In a systemwith multiple users reviewing the interim of final results, it could bedone through a website interface where in exchange for the reviewing ofthe data the users were given access to the peer reviewed articles orthe input data at no charge.

Another embodiment of the AI system 104 may be constructed in a way toquestion data points and how it affects the entire algorithm. Forexample, one looks at entire chaining of information to end up with aconclusion. For example, if one looks at a research article, theconclusions of a research article are often based upon the referenceswithin the article. However, if one of these references is erroneous, itwould be necessary to remove this data and through machine learningchange the ultimate algorithm so that the conclusion is changed based onchanging or altering one of the reference or data points. The operatorcould change this data format or this reference as and mark it as aninvalid or questionable point. Grouping AI algorithms could also be usedto do this. This method of interim analysis could also be used to allowand experts to review and weight the results for search results that maynot have a black and white or definitive answer. The expert, or anexpert system algorithm, could be used to weight the output of the AIsystem 104 or search results. This could be used in internet searchengines, drug databases, or any algorithm that produces none definitiveresults.

For example, if one changes the data point/reference of how a black malewould function relative to a total knee replacement versus elderly whitefemale. The AI system 104 would allow for changes to one of the datapoints in terms of functional return or risks of keloid formation andthe impact on how this affects stiffness of the joint, range of motion,and function. It may have an impact on the algorithm for sensing theligament balance within the joint or how one would allow bone resectionvia MAKO robotic system to move the knee. The AI system 104 would allowthe user to alter that based on the risks of scar tissue forming andwhat would the scar risk be for elderly white female versus youngerblack male versus a patient with sickle-cell anemia versus patient thatwould have very elastic soft tissue. Current systems do not allow thischange in concepts on the fly based on individual data. This could beinputted manually by the operator or it could allow multiple variablesto say if the patient has sickle-cell, Ehlers-Danlos, or keloidformation. These changes of the data could affect the incision approach,robotic mechanism for tissue resection, tissue repair, and the amount ofbone to be removed for a total knee replacement to optimize function.This would also link to sensors and postoperativefunction/rehabilitation so one could enhance therehabilitation/recovery. If this patient needs more aggressive therapyto work on flexion or to deal with keloid/hyperelasticity of the tissueor how one could improve scar formation and function.

Embodiments described herein may be implemented using global expertsystem 112, which utilizes the knowledge from one or more experts in thealgorithm executed thereby. A system of rules and data is required priorto the running of global expert system 112. Global expert system 112 canbe implemented such that in the introduction of new knowledge is rebuiltinto the code, or the code can dynamically update to include newknowledge generated by global expert system 112 and/or AI system 104. Inan embodiment, updates to the code of global expert system 112 arevalidated with respect to regulatory requirements before implementation.

In an embodiment, AI system 104 implements one or more geneticalgorithms. Genetic algorithms use the principles of natural selectionand evolution to produce several solutions to a given problem. In anexemplary approach, AI system 104 randomly creates a population ofsolutions of a problem. The AI system 104 then evaluates and scores eachsolution using criteria determined by the specific application. The AIsystem 104 selects the top results, based on the score, and uses them to“reproduce” to create solutions which are a combination of the twoselected solutions. These offspring go through mutations and AI system104 repeats these steps or a portion of the steps until a suitablesolution is found. Additionally or alternatively, AI system 104 utilizesother known AI techniques such as neural networks, reinforcementlearning, and the like.

In an aspect, AI system 104 uses a combination of known AI techniques.In an embodiment, AI system 104 uses an expert system (e.g., globalexpert system 112) as a global ruleset that has a local copy of therules created for each patient who is checked into system 100. AI system104 uses adaptive rules to modify the local rule set for each individualpatient. Instead of a complete local copy, only the modified rules couldbe kept locally at a computing device executing processor-executableinstructions for implementing AI system 104 and/or global expert system112 to reduce the required memory needed. In an embodiment, a safetycontrol is included so that rules and/or alerts generated by AI system104 and/or global expert system 112 (e.g., the inference engine) cannotbypass one or more (or a group) of the global or expert system rules.Rules generated by AI system 104 or rules from the predictive rules arecompiled and/or integrated into the global or expert rule set inaccordance with one or more embodiments.

One embodiment of using system 100 in a hospital environment includesmaximizing a patient's ability to rest at night by scheduling certainprocedures and activities around the patient's sleep patterns. During anormal sleep pattern a person goes through different cycles of sleep,including light sleep, deep sleep, and REM. A patient's sleep patterns,heart rate, and movements at night are monitored using patient monitorsensors 102 (e.g., Fitbit® activity tracker, Apple® Watch, smartphonecomputing device, an electronic ring, or any commercial or customtracker with the ability to record and or transmit patient's movementsand sleep patterns). Additionally or alternatively, system 100 utilizespatient monitor sensors 102 in the form of existing sensors used inhospital monitoring systems, hospital records, pulse oximeter, retinalchanges, temperature, or thermal gradients, changes in diet or foodpatterns from dieticians, and mediation from the pharmacy. The datacollected from patient monitor sensors 102 is then uploaded (e.g., via acommunications network) to AI system 104 in real time for analysis.Additionally or alternatively, the data collected from patient monitorsensors 102 is manually uploaded to AI system 104 for analysis. The AIsystem 104 then uses this information to ensure that the patient is inthe correct sleep cycle when they must be woken up for procedures, suchas by execution of a waking/alerting algorithm 200 (FIG. 2 ). Forexample, system 100 monitors all patients on a certain a floor andcreates an optimized map or order of blood draws for the phlebotomist tominimize the patients being disturbed from deep sleep. This map could beprinted along with the ordered bloodwork or could be sent wirelessly tohealthcare provider device 120 (e.g., a tablet or smartphone computingdevice) and updated in real time. The same algorithm could be used bymultiple departments in the hospital so that medication delivery, food,etc. is optimally scheduled.

For patients who are taking medications at night outside of the hospitalthe waking/alerting algorithm 200 may be implemented by one or morepatient devices 122 (e.g., a smartwatch, smartphone computing device,tablet computing device, or other monitor) to wake the user at anoptimal time in the sleep cycle to take medication. In an embodiment,system 100 is also used to determine the optimal time of the day to takea medicine for an individual user based on daily activity level, sleeppatterns, metabolism, and like factors.

FIG. 2 illustrates an exemplary embodiment of the waking/alertingalgorithm 200. In an embodiment, the waking/alerting algorithm 200comprises processor-executable instructions embodied on a storage memorydevice of a computing device to provide waking/alerting techniques formedication delivery via a software environment. For example, thewaking/alerting algorithm 200 may be provided as processor-executableinstructions that comprise a procedure, a function, a routine, a method,and/or a subprogram utilized independently or in conjunction withadditional aspects of system 100 according to an exemplary embodiment ofthe disclosure. In an embodiment, the waking/alerting algorithm 200 isexecuted by a computing device, such as one or more of a computingdevice implementing AI system 104, patient monitor sensors 102, andpatient devices 122 in accordance with one or more embodiments of thedisclosure.

At 202, the patient or healthcare provider enters the medicationschedule. For example, the patient may enter the medication schedule viapatient monitor sensors 102, patient device 122, and/orpatient-controlled mediation delivery subsystem 114 and the healthcareprovider may enter the medication schedule via healthcare providerdevice 120.

At 204, the computing device determines whether a dose is required atthe current time according to the entered schedule. When a dose isdetermined to be not required at 204, the algorithm 200 loops back to204. When a dose is determined to be required at 204, the algorithmadvances to 206.

At 206, the computing device determines whether the user (e.g., patient)is awake. In an embodiment, the computing devices uses data collectedfrom one or more patient monitor sensors 102 to make this determinationas further described herein. Exemplary patient monitor sensors aredescribed herein, including a smartwatch, an activity monitor, a camera,and the like. When the user is determined to be awake at 206, the useris alerted at 208 (e.g., via patient device 122 and/or patient monitorsensors 102) for medication delivery. After alerting the user at 208,the algorithm advances back to 204. When the user is determined to benot awake at 206, the algorithm advances to 210.

At 210, the computing device determines whether the user is in lightsleep. In an embodiment, the computing devices uses data collected fromone or more patient monitor sensors 102 to make this determination asfurther described herein. Light sleep includes sleep that falls into thecategories of Stage 1 and Stage 2 in accordance with an aspect of thedisclosure. When the user is determined to be in light sleep at 210, theuser is alerted at 208 (e.g., via patient device 122 and/or patientmonitor sensors 102) for medication delivery. After alerting the user at208, the algorithm advances back to 204. When the user is determined tonot be in light sleep at 210, the algorithm advances to 212.

At 212, the computing device determines whether the user is in deepsleep and/or rapid eye movement (REM) sleep. In an embodiment, thecomputing devices uses data collected from one or more patient monitorsensors 102 to make this determination as further described herein. Deepsleep includes sleep that falls into the categories of Stage 3 and Stage4 in accordance with an aspect of the disclosure. When the user isdetermined to not be in deep or REM sleep at 212, the user is alerted at208 (e.g., via patient device 122 and/or patient monitor sensors 102)for medication delivery. After alerting the user at 208, the algorithmadvances back to 204. When the user is determined to be in deep and/orREM sleep at 212, the algorithm advances to 214.

At 214, the computing device determines whether a maximum wait time isexceeded. For example, the maximum wait time may be a predefinedthreshold for the maximum time allowed between medication dosagedeliveries. When the maximum wait time is not exceeded at 214, thealgorithm loops back to 206. When the maximum wait time is exceeded at214, the user is alerted at 208 (e.g., via patient device 122 and/orpatient monitor sensors 102) for medication delivery. After alerting theuser at 208, the algorithm advances back to 204.

In an embodiment, the waking/alerting algorithm 200 executes on acomputing device as a standalone system. In another embodiment, thewaking/alerting algorithm 200 executes on a computing device as part ofa larger (e.g., hospital-wide) system. Exemplary computing devices onwhich the waking/alerting algorithm 200 can be executed include, but arenot limited to, a smartwatch, an activity monitor, a smartphone, and thelike. In an embodiment, the user is alerted via the device that executesthe waking/alerting algorithm 200 (e.g., a smartwatch, an activitymonitor, a smartphone, etc.). In another embodiment, the user is alertedvia an external device, such as a pager, an alarm clock, a notificationgiven to a healthcare provider or other caregiver, and the like.

The alerting algorithm 200 is not limited to sleep and can be used whenthe user is performing other activities (e.g., work, personal,entertainment, employment). For example, the alerting algorithm 200 canbe executed on a portable computing device, such as a smart watch, whilethe user is running or exercising. Similar to tracking a user's sleep,the alerting algorithm 200 can track the user's exercising and then thealgorithm would alert the user to take the medication, via the portablecomputing device, after the user has finished exercising or the maximumwait time is exceeded. In addition, similar to light and deep sleep, thealerting algorithm 200 can determine if the user is experiencing lightexercise or heavy exercise and alert the user accordingly regarding themedication. Moreover, the alerting algorithm 200 can include GPS andalert the user to take a medication once the user has reached a specificlocation, such as their home.

Referring again to FIG. 1 , system 100 may be integrated into devices(e.g., patient monitoring devices 102, patient devices 122, etc.) in aroom that can disturb the patient during sleep. The system 100 may be acentral system or a local version thereof in accordance with one or moreembodiments. An example includes automated blood pressure measurements(e.g., automated blood pressure measurement algorithm 300; FIG. 3 )using a sphygmomanometer. By monitoring the patient's sleep pattern, themeasurements are performed when the patient is in light sleep tomaximize the amount of deep sleep the patient has. In other words, lightsleep stages are interrupted while deep sleep stages are leftuninterrupted. In an embodiment, AI system 104 executes a learningalgorithm to determine if the patient is a heavy sleeper or a lightsleeper and if taking the measurements during a deep sleep cycle did notdisturb the patient they could be performed during those cycles. Inanother embodiment, system 100 puts hard limits on the amount of time ameasurement can be shifted forward or backward to ensure that too muchtime does not elapse between measurements.

FIG. 3 illustrates an exemplary embodiment of an automated bloodpressure measurement algorithm 300. In an embodiment, automated bloodpressure measurement algorithm 300 comprises processor-executableinstructions embodied on a storage memory device of a computing deviceto provide automated blood pressure measurement techniques via asoftware environment. For example, the automated blood pressuremeasurement algorithm 300 may be provided as processor-executableinstructions that comprise a procedure, a function, a routine, a method,and/or a subprogram utilized independently or in conjunction withadditional aspects of system 100 according to an exemplary embodiment ofthe disclosure. In an embodiment, the automated blood pressuremeasurement algorithm 300 is executed by a computing device, such as oneor more of a computing device implementing AI system 104, patientmonitor sensors 102, and patient devices 122 in accordance with one ormore embodiments of the disclosure.

At 302, the patient or healthcare provider enters a blood pressuremeasurement schedule and/or patient sleep schedule. For example, thepatient may enter the blood pressure measurement schedule and/or patientsleep schedule via patient monitor sensors 102, patient device 122,and/or patient-controlled mediation delivery subsystem 114 and thehealthcare provider may enter the blood pressure measurement scheduleand/or patient sleep schedule via healthcare provider device 120.

At 304, the computing device determines whether a blood pressuremeasurement is due at the current time according to the enteredschedule. When a measurement is determined to not be due at 304, thealgorithm 300 loops back to 304. When a measurement is determined to berequired at 304, the algorithm advances to 306.

At 306, the computing device determines whether the user (e.g., patient)is awake. In an embodiment, the computing devices uses data collectedfrom one or more patient monitor sensors 102 to make this determinationas further described herein. Exemplary patient monitor sensors aredescribed herein, including a smartwatch, an activity monitor, a camera,and the like. When the user is determined to be awake at 306, a bloodpressure measurement is automatically taken at 308. After taking themeasurement at 308, the algorithm advances back to 304. When the user isdetermined to not be awake at 306, the algorithm 300 advances to 310.

At 310, the computing device determines whether the user is in a lightsleep or a heavy (i.e., deep) sleep. In an embodiment, the computingdevices uses data collected from one or more patient monitor sensors 102to make this determination as further described herein. Light sleepincludes sleep that falls into the categories of Stage 1 and Stage 2 andheavy sleep includes sleep that falls into the categories of Stage 3 andStage 4 and/or REM sleep in accordance with an aspect of the disclosure.When the user is determined to be in light sleep at 310, the algorithmadvances to 312. When the user is determined to be heavy sleep at 310,the algorithm advances to 316.

At 312, the computing device determines whether the user is in lightsleep or awake. In an embodiment, the computing devices uses datacollected from one or more patient monitor sensors 102 to make thisdetermination as further described herein. When the user is determinedto be in light sleep or awake at 312, a blood pressure measurement isautomatically taken at 308. After taking the measurement at 308, thealgorithm advances back to 304. When the user is determined to not be inlight sleep or awake at 312, the algorithm advances to 314.

At 314, the computing device determines whether a maximum timeout isexceeded. For example, the maximum timeout may be a predefined thresholdfor the maximum time allowed between blood pressure measurements. Whenthe maximum timeout is not exceeded at 314, the algorithm loops back to312. When the maximum timeout is exceeded at 314, a blood pressuremeasurement is automatically taken at 308. After taking the measurementat 308, the algorithm advances back to 304.

At 316, the computing device determines whether the user is in heavy orREM sleep. In an embodiment, the computing devices uses data collectedfrom one or more patient monitor sensors 102 to make this determinationas further described herein. When the user is determined to be in heavyor REM sleep at 316, the algorithm advances to 318. When the user isdetermined to not be in heavy or REM sleep at 316, a blood pressuremeasurement is automatically taken at 320 before proceeding to 322.

At 318, the computing device determines whether a maximum timeout isexceeded. For example, the maximum timeout may be a predefined thresholdfor the maximum time allowed between blood pressure measurements. Whenthe maximum timeout is not exceeded at 318, the algorithm loops back to316. When the maximum timeout is exceeded at 318, a blood pressuremeasurement is automatically taken at 308 before proceeding to 322.

At 322, the computing device determines whether the automated bloodpressure measurement taken at 320 awakened the user. In an embodiment,the computing devices uses data collected from one or more patientmonitor sensors 102 to make this determination as further describedherein. When the user is determined to not be awakened at 322, thealgorithm advances back to 304. When the user is determined to beawakened at 322, the computing device sets the user sleep state to“light” at 324 before advancing back to 304.

In an embodiment, the automated blood pressure measurement algorithm 300executes on a computing device as a standalone system. In anotherembodiment, the automated blood pressure measurement algorithm 300executes on a computing device as part of a larger (e.g., hospital-wide)system. Exemplary computing devices on which the automated bloodpressure measurement algorithm 300 can be executed include, but are notlimited to, a smartwatch, an activity monitor, a smartphone, and thelike. While this embodiment utilizes a blood pressure monitor, it isunderstood that this system could be used for any automated test whichis done during sleep. Moreover, it is understood that this embodimentcould be used in non-medical applications where the interruptingactivity might disturb sleep patterns such as, but not limited to, theoperation of robotic vacuums, cleaning systems, air filters, heatingventilation and air conditioning systems, non-emergency alarms, washersand dryers, and/or dishwashers.

Referring again to FIG. 1 , aspects of system 100 are integrated intothe bedside call button (e.g., call button 116), in an embodiment. Whenthe patient presses call button 116, a signal is sent to the healthcareprovider (e.g., nurse, etc.) via a traditional call system and/orthrough smart alert system 118. A predictive alert subsystem of smartalert system 118 records one or more conditions when call button 116 waspressed (recent food, movement, sleep patterns, BP, pulse, sweat, skinresistance, etc.). The smart alert system 188 the uses this informationto start to compile an adaptive system that is capable of alerting thehealthcare provider (e.g., via healthcare provider devices 120) beforethe patient presses call button 116 in the future. For example, whensmart alert system 118 determines that a similar set of conditions aremet or the conditions are within a threshold, smart alert system sendsthe alert to the healthcare provider via healthcare provider devices120. In an embodiment, after the healthcare provider responds to asystem-generated call button press, the healthcare provider and/or thepatient have the ability to score the alert. For instance, thehealthcare provider may score the alert via healthcare provider devices120 and the patient may score the alert via patient monitoring sensors102 and/or patient devices 122. False alerts would receive a low ratingand reduce the chance of a similar condition set triggering an alert inthe future. A higher rating, or a reduction of call button 116 pressesby the patient, could indicate a positive result and increase the chancethat similar conditions would trigger an alert to the healthcareprovider via healthcare provider devices 120. In an aspect, smart alertsystem 118 uses this scoring system and/or other known AI techniques tointelligently modify the rule set (e.g., rules in global expert system112 and/or AI system 104) for each patient. In another embodiment,alerts from multiple patients are weighted by severity of the situation(e.g., higher priority given to more critical conditions, etc.) toensure the healthcare providers are able to prioritize emergencysituations.

In an embodiment, aspects of system 100 are integrated into medicationsdelivered by patient-controlled medication delivery subsystem 114 (e.g.,patient controlled analgesia (PCA), etc.). In an aspect, the AI system104 monitors one or more physical properties of a patient, such aspatient sleep patterns, pulse rate, blood pressure, skin resistance, andlike properties, to determine when a patient's pain has increased. Bypredicting pain levels, AI system 104 then varies the dosing so that thepatient's discomfort is minimized. In another aspect, the AI system 104determines when a patient is pressing the button but not experiencingpain and adjusts the dosage accordingly to minimize abuse.

FIG. 4 is a block diagram of an exemplary blood draw algorithm andsystem. In an aspect, patient monitor sensors 102 sense physicalproperties associated with one or more patients. The data collected bypatient monitor sensors are transmitted to AI system 104 via acommunications network (e.g., wired communication, wirelesscommunications, etc.), as further described herein. The data collectedby patient monitor sensors may additionally or alternatively betransmitted via the communications network to a data repository (e.g.,database, cloud service, etc.) for storing the data in advance of AIsystem 104 utilizing the data. At 402, AI system 104 detects andcompiles patient sleep patterns from the collected data. At 404, bloodwork scripts are entered into AI system 104 and/or a data repositoryassociated therewith. For example, the blood scripts may be entered by ahealthcare provider via healthcare provider devices 120, obtained fromelectronic medical records database 110, and the like. At 406, AI system104 executes an intelligent scheduling algorithm to generate anoptimized and/or personalized blood draw procedure schedule for eachpatient and/or for the group of patients. At 408, the generated scheduleand/or real-time updates thereto are communicated to one or morehealthcare providers (e.g., phlebotomist, etc.). For example, thegenerated schedule and/or real-time updates are communicated to thehealthcare provider via healthcare provider devices 120 (e.g., tabletcomputing device, etc.), as further described herein. In an embodiment,one or more aspects of the exemplary blood draw algorithm and/or systemare integrated with a distributed medical record (DMR) system and/orlaboratory software.

FIG. 5 illustrates an exemplary architecture of a computing device 500configured to provide aspects of the systems and processes describedherein via a software environment. In this embodiment, the computingdevice 500 includes a processor 502, a memory device 504, aninput/output (I/O) interface 506, and a display interface 508. Thememory device 504 includes processor-executable instructions forexecuting by processor 502 that carry out processes for artificialintelligence techniques that optimize and/or personalize one or moreaspects of monitoring, diagnosis, treatment, and prevention of disease,illness, injury, physical and/or mental impairments of the patient,emulate decision-making abilities of one or more human experts regardingone or more aspects of monitoring, diagnosis, treatment, and preventionof disease, illness, injury, physical and/or mental impairments of thepatient, alert healthcare providers before the patient presses callbutton 116, provide waking/alerting techniques for medication delivery,and provide automated blood pressure measurement techniques, as furtherdescribed herein. In this manner, computing device 500 comprises aspecial-purpose computing device for optimizing and/or personalizing oneor more aspects of monitoring, diagnosis, treatment, and prevention ofdisease, illness, injury, physical and/or mental impairments of thepatient, emulating decision-making abilities of one or more humanexperts regarding one or more aspects of monitoring, diagnosis,treatment, and prevention of disease, illness, injury, physical and/ormental impairments of the patient, alerting healthcare providers beforethe patient presses call button 116, providing waking/alertingtechniques for medication delivery, and/or providing automated bloodpressure measurement techniques in accordance with an aspect of thedisclosure.

The processor 502, memory 504, I/O interface 506, and display interface508 are communicatively connected and/or electrically connected to eachother. The I/O interface 506 may be communicatively connected to acommunications network, as further described herein, an I/O device, andthe like. The processor 502 is adapted to execute processor-executableinstructions stored in the memory 504 for optimizing and/orpersonalizing one or more aspects of monitoring, diagnosis, treatment,and prevention of disease, illness, injury, physical and/or mentalimpairments of the patient, emulating decision-making abilities of oneor more human experts regarding one or more aspects of monitoring,diagnosis, treatment, and prevention of disease, illness, injury,physical and/or mental impairments of the patient, alerting healthcareproviders before the patient presses call button 116, providingwaking/alerting techniques for medication delivery, and/or providingautomated blood pressure measurement techniques in real time. The I/Ointerface 506 of FIG. 5 provides a physical data connection betweencomputing device 502 and a communications network, in an embodiment. Inan embodiment, I/O interface 506 is a network interface card (NIC) ormodem. The display interface 508 provides a physical data connectionbetween computing device 502 and a display device. In an embodiment, thedisplay device is an LCD indicator display that includes a keypad forhuman interface. In another embodiment, the display device is atouchscreen on computing device 502 and/or a touchscreen of asmartphone, tablet computing device, or the like comprising one or moreof patient monitoring sensors 102, patient-controlled medicationdelivery subsystem 114, call button 116, healthcare provider devices120, and patient devices 122.

In addition to using artificial intelligence in the healthcareenvironment, virtual reality (VR) and/or augmented reality (“VR/AR”)games can play an important part in healthcare and rehabilitation.Virtual reality creates a convincing simulated 3D environment whichallows the user to interact in a natural way for example, by projectingvisible light directly onto the retina to create a virtual image.Virtual reality systems are traditionally classified as immersive,semi-immersive, and non-immersive systems. In an exemplary immersive VRsystem, the user wears a head mounted display (HMD). Sensory inputs aredelivered through the HMD unit, gloves, suits, clothing, and the like.Additional movement and responses may be obtained with other handheldand/or body sensors, cameras, joysticks, and the like. In an exemplarysemi-immersive system, multiple monitors and/or large screen projectorswith a wide field of view are used. In an exemplary non-immersivesystem, conventional monitors, gaming platform, controllers or keyboardsand a mouse are used. Embodiments described herein can be implementedusing an immersive, semi-immersive, and/or non-immersive system.

During physical therapy, VR can be used to create an environment whereexercises and therapy can be personalized to an individual patient'sneeds. FIG. 6 illustrates an exemplary VR system in accordance with anaspect of the disclosure. The VR system includes a Joint Active System(JAS) device 602, a HMD 604, a cuff 606, sensors 608 in the form ofcameras in this embodiment, and a computing device 610 (e.g., patientdevice 122).

In one embodiment, a static progressive stretch unit such as a JointActive System (JAS) device 602 is integrated with sensors (e.g., patientmonitor sensors 102) which would then be used as inputs to the VRsystem. The static progressive stretch unit may also be a dynamicsplinting device, tens device, electro-stimulation device, ultrasounddevice, exercise bike, continuous passive motion device (CPM), bonegrowth stimulator, and any other therapy device. In an embodiment, thesensors are integrated into cuff 606, attached to the patient, and thelike. The JAS device 602 could be manually driven or equipped with amotor to drive the movement. Traditionally, JAS devices are used in ahome use environment. As the patient worked through the treatmentprotocol for the JAS device the virtual environment could appear to bein a physical therapy office, for example. A virtual physical therapistcould instruct the patient through the stretch and hold protocol, in anembodiment. In addition to monitoring the position on the device, thesystem could monitor other inputs such as blood pressure, heart rate,and the like. The virtual therapist could use aspects of system 100(e.g., AI system 104, etc.) to dynamically adjust the treatment inresponse to the patient's ability for movement and/or feedback fromother sensors. In addition to monitoring the position of the JAS device602 by using sensors, the system could use a camera and/or other methodsto monitor the position of patient to determine the degree of stretch.Instead of using the virtual environment of a physical therapy office,any environment could be used that could enhance therapy, such as byrelaxing the patient, motivating them to perform therapy, and the like.Instead of the patient or therapist selecting the virtual environment,one could be elected from existing patient information or web searchhistory. Some forms of physical therapy and stretching could be donewithout an external device and could use gloves, position sensors,cameras, and the like to monitor the patient's position and move themthrough stretches. In an embodiment, the sensors and motor drive may beattached to standalone computing device 610. In another embodiment,processing is performed in HMD 604. In another embodiment, processing isintegrated into JAS device 602. Although FIG. 6 illustrates an exemplaryembodiment in which the elbow is used, the VR system can be used on anyjoint or extremity. Moreover, sensors 608 may include multiple cameras,infrared cameras, thermal cameras, electromagnetic cameras, and thelike.

Other controllers include those created specifically designed for thepurpose of physical rehabilitation. The devices are designed to provideresistance and/or actively stretch the treatment site, as illustrated bydevice 700 in FIG. 7 . This is done with motors, magnets,electromagnets, pneumatics, hydraulics, piezoelectric breaks, and/orlike methods. For example, the device may be used as a pressurecontroller, where the patient actively moves and the device providesresistance. In an aspect, the device is actively driven by the VR systemto manipulate the joint or body part under therapy.

In an embodiment, the device is a multi-axis controller attached to thearea being treated, as illustrated in FIG. 8 . For example, multiplemanipulators 802, 804 may be attached to the same joint. The degree offreedom would be based on the joint being manipulated, for example. Inan embodiment, arm (e.g., manipulators 802, 804) movement is linked. Inanother embodiment, arm movement is independent.

FIG. 9 illustrates an embodiment of aspects of the VR system in which aplurality of controllers 902, 904 are used in combination. Asillustrated, one controller 902 is coupled to a lower leg portion of thepatient and another controller 904 is coupled to an upper leg portion ofthe patient. In an embodiment, sensors (e.g., position sensors, patientmonitor sensors 102, etc.) are embedded in one or more of thecontrollers. In another embodiment, position is obtained from externalsensors 608 (e.g., cameras, patient monitor sensors 102, etc.).

FIG. 10 illustrates an embodiment of aspects of the VR system in whichshape memory, alloy, polymers, electroactive polymers (EAP),piezoelectric elements and the like are embedded into one or morearticles of clothing 1002 and used with the VR system to provideresistance and/or manipulate joints. In an embodiment, the patient wearsclothing articles with memory fiber and the like that coach and/or forcethe patient through the correct posture. In another embodiment, thecontroller is moved in a pattern that forces the patient through aseries of stretches while the visuals of the virtual reality (e.g., HMD604) shown to the patient are designed to calm the patient, promotestretching and/or compliance, and the like.

FIG. 11 illustrates an embodiment of VR system including a controller1102 that allows the patient to manipulate a device with the patient'shands. Depending on the sensors, the controller 1102 can simulatedifferent items. If the therapy was for occupational therapy, forexample, the VR system may operate to enable the patient to think theyare lifting a box. In an embodiment, different weights are simulated byusing the resistance of the motors in the controller 1102. In anotherembodiment, the device includes a series of electrical windings that canbe enabled at various strengths. When therapy and/or occupationalmedicine is performed on a metal plate, the windings are energized atvarious levels, creating a simulation of lifting varying size and/orweight objects. The VR system adjusts the perceived weight of the objectby varying the magnetic field, in an embodiment. In an embodiment, thewindings are placed in the floor on the plate in which the therapy takesplace. The VR system can also monitor the position of the user's body,using one or more external sensors 608, to ensure the occupationaltherapy is being performed properly by the user. The VR system may thenbe able to instruct the user to correct the position of their bodyaccordingly.

FIG. 12 illustrates an embodiment of the VR system in which a platform1202 is made with windings 1204, 1206 (e.g., wire coils) in the floorfor stretches and/or exercises involving lifting. In an embodiment, theamount of electrical current flowing through the windings is varied withthe virtual content displayed to the patient via HMD 604. In anembodiment, the controller varies the current in time with the displayedcontent to ensure the patient/user perceives body ownership. In anembodiment, the object 1208 (e.g., box, etc.) to be lifted, which may bethe controller in one or more embodiments, is made of a material that isaffected by the magnetic field generated by the windings 1204, 1206. Inanother embodiment, the magnetic field is reversed to repel the object1208. In yet another embodiment, shoes and/or articles of clothing wornby the patient are comprised of materials that can bemanipulated/affected by the magnetic field. In an embodiment, the VRsystem utilizes object 1208 and/or the magnetic field generated bywindings 1204, 1206 to simulate different weights, textures, positions,forces by a therapist, and the like.

In addition to monitoring the user's heart rate and blood pressurethrough traditional methods, the user's retinas are monitored using acamera mounted in the HMD 604, in an embodiment. In this embodiment,changes in blood flow and dilation of the retinas are used to determineblood pressure. In addition to blood pressure, the system is capable ofmonitoring excitement, sadness, other emotions, response to pain, andthe like. Monitoring these reactions could be used to pace conversationsand augment interactions with virtual constructs in an AI environment(e.g., AI system 104). A user's negative response to a construct wouldbe a trigger for the construct to change pace, pitch, tone of theconversation, and the like. This would allow the user to have a morerelaxed interaction with aspects of the system. Sensors, as furtherdescribed herein, are integrated into the eye surrounds and the headstrap of the HMD, in accordance with one or more embodiments. The bloodpressure readings could be an absolute measurement or a differentialmeasurement. These measurements could be taken in real time and changesin blood pressure and/or heart rate could be used by the AI system 104to monitor and diagnose conditions. In addition to the using the directmeasurements, changes in blood pressure and/or pulse in response tocertain questions and/or other stimuli could indicate a false answer(e.g., prescription drug abuse, etc.).

In addition to the use of systems and methods described herein inmedical applications, the described systems and methods may also be usedfor direct marketing purposes and the like. By monitoring the user'sresponse to video, virtual reality, augmented reality being watched, andthe like, AI system 104 creates a profile for the user. Positiveresponses to stimuli would be recorded and content and/or commercialsare created based off the user's profile. For example, if a user had apositive reaction to cars in general, then content or storylines forcars could be used. This could work on a more specific level, such as ifthe user had a positive reaction to a certain brand of car then thevideo content could be modified to use more of that brand, or commercialfor local dealers of that brand could be displayed to the user. Thisinformation may also be used to recommend new content for the user towatch. In addition, a response to a commercial advertisement may triggermore information to be sent to the user via phone, email, traditionalmail, and the like. The information may also be used to create adatabase of user interests.

In an embodiment, differential blood pressure is measured using anultrasound device placed on a user's finger and/or a stationary devicesimilar to a fingerprint reader, for example. In addition to thepressure, other attributes of the blood flow are monitored, such asturbidity rate using Doppler, and the like. These sensors may beintegrated into a watch or ring, as further described herein, forconstant monitoring and the information may be stored and/or transmittedfor further analysis (e.g., by AI system 104) or the information may beanalyzed (e.g., by AI system 104) in real time.

Other aspects of the disclosure provide a home therapy device topatients struggling with vision related diseases (e.g., Age-RelatedMacular Degeneration, Histoplasmosis, Cataracts, Diabetic, etc.). Thisdevice will track progress and adapt the test to best fit each patient'sdisability. In an embodiment, this device includes HMD 604 and/or otheraspects of the VR system in communication with AI system 104 and/orother aspects of system 100. The eye monitoring and vision enhancementsystems and methods described below are exemplary and non-limiting.

In an embodiment, this device is a home-use device that will fit overthe user's face to do routine eye exams and/or provide crucial and/orregular monitoring of the eyes. The device uses software that adapts tothe patient's need and will provide regular reports to the user via aportable display, PC, and/or cell phone app. The device may also providedirect feedback to the healthcare provider (e.g., ophthalmologist,etc.). This feedback allows the healthcare provider to monitor theprogress of a treatment and/or to inform the provider of negativesetback that could require immediate attention.

The device, in an embodiment, is comprised of a goggle-like device thatis placed over the eyes, a display, and a mouse/pad. The goggle devicedisplays information to the patient in the central and peripheralregions. Software would run through a series of tests (e.g., AmslerGrid, Automated Static Perimetry, Non-Contact Tonometry test, Eye VisionExam, Optical coherence tomography (OCT), etc.) based on the patient'sdiagnosis to monitor and track the patient's progress. During certaintests, the user gives feedback. The feedback will be generated throughthe mouse/pad and/or other input device and sent to the goggles via wireand/or wireless communication channels. One example would be during theAmsler Grid test the user would be required to indicate which areas ofthe grid are distorted and which parts of the grid are actually missingby drawing a line with the mouse/pad. Another example would be duringthe Automated Static Perimetry exam the user would be required to pressor tap the mouse/pad to indicate when a light can be seen. The testresults are compiled into a report that is displayed to the user and/orsent to the healthcare provider.

This device enables regular monitoring of eye related diseases. Eyerelated diseases are often complex and difficult to know when treatmentsare not working as expected. This device enables users to keep recordsof progress in terms of days and weeks instead of months. Doctorappointments are pricey and specialists (e.g., ophthalmologists) areoften difficult to get immediate appointments. Patients struggling withAMD are often provided with an Amsler grid to monitor progress of thedisease but many times it is difficult to track progress from day to dayusing a paper sheet and there is variability of results due to distancefrom the sheet and room lighting. Many times the patient doesn't realizea change in vision until the next appointment and by then the bleed orscaring in the back of the eye could be pretty significant.

Regular monitoring of the eyes. In an aspect, this device allows theuser to track progress of their vision on a regular basis. Many timesusers meet with the healthcare provider on a monthly basis or even worseonly several times a year. Between these appointments, the device wouldmonitor and track the progress of vision and inform the patient and/orhealth provider when a more immediate appointment is required.

Integration of multiple tests. In an aspect, this device integratesmultiple tests into the goggle software (e.g., Amsler Grid, AutomatedStatic Perimetry, Non-Contact Tonometry test, Eye Vision Exam, Opticalcoherence tomography (OCT), etc.). Conventional cellular phoneapplications allow users to test and track their Asmler Grid test andEye Vision Exam, but these applications lack constancy (e.g., distanceof phone from patient, etc.) and don't adjust to each patient's needs(e.g., based on eye disease and progression, etc.).

Additional Enhancements. The goggles may also be used to enhance vision.Magnify. In an aspect, the user uses the goggles to magnify objects theyare looking at by using a camera lens and displaying the object largeron the display. Blurry Vision. Some diseases cause blurry vision due toprotein buildup that clouds the eye lens. In an aspect, this deviceenables the patient to adjust the image on the goggle display to includeoutlines that define objects in the room. Though in these cases visioncannot be restored, this device would help patients function in theirhousehold. Scotoma (defect within a field of vision—blind spot). In thecase of the patient suffering from AMD, the patient may suffer fromscotoma (FIG. 13 ), a partial loss of vision, or a blind spot in anotherwise normal visual field. FIG. 13 is an image of an Amsler Gridshowing a large scotoma encroaching on the central fixation with somedistortion. In an aspect, this device would require the user to indicatethe scotoma area. Once the area is determined, the camera data isadapted to allow that area of data to be moved into the peripheral fieldto allow the user calibrate based off the eye test report and displaythe image to best suit the patient's needs. In a case where the centralvision is missing, the image in the central vision may be displayed inthe peripheral location (FIG. 14 ). FIG. 15 is an image of ducks with ascotoma affecting the vision of the duck on the left side. FIG. 16 is amodified image after visual data is manipulated in accordance withsystems and methods described herein with the patient's scotoma data.

Distortion Filter. In the case of histoplasmosis scaring, the vision canbe distorted as seen in FIG. 18 . The distortion is caused from theheight of the scar (FIG. 17 ). In an embodiment, the user adjusts theAsmler grid lines until the lines are straight so that the algorithm ofthe display distorts the camera data in such a way that the user seesclearly. For example, the patient may select a gradient tool and focusthe mouse/pad over the distorted area and make adjustments to distortthe image back to the straight line grid (FIG. 19 ). In this manner, thecamera data is filtered to adjust the image to counter the distortioncaused by the scar. Additionally or alternatively, results from thedistortion test in accordance with an aspect of the disclosure are usedto generate lenses for corrective glasses that compensate fordistortion. FIG. 20 is a flowchart of an exemplary algorithm forfiltering camera data to adjust the image to counter the distortioncaused by the scar. At 1302, the computing device executing thealgorithm grabs the video input. At 1304, the computing device grabs thepatient's distortion configuration. At 1306, the computing deviceexecuting the algorithm applies a distortion filter using the patient'sconfiguration. At 1308, the computing device outputs the filtered videoto a display device for viewing by the patient.

In an embodiment, the eye monitoring and vision enhancement deviceincludes a goggle device 1350, an external display 1352, and a mouse/pad1354 for user feedback, as illustrated in FIGS. 22 and 23 . In anaspect, the goggle device 1350 includes a display 1356 that fully coversthe wearer's (e.g., patient's) eyes to provide an image to the patientin the central and peripheral vision of the patient. The goggles 1350use software that is configured to perform one or more tests based onthe patient's diagnosis. Each test will follow the patient's individualprogression of the eye disease and adjust (e.g., optimize, personalize,etc.) the test to best fit the patient's needs. The goggles 1350 alsoinclude camera lenses 1358 that provide visual data to the controller(e.g., processor) in the goggle device where the visual data will bemanipulated based on the needs of the patient (e.g., improve image,magnify image, etc.). In one embodiment, the device 1350 may alsoinclude one or more thermal sensors configured to detect the possibleonset of infection.

Display 1356: In an embodiment, the display comprises a tablet computingdevice, a laptop computing device, a cellular phone, a smartphone, adisplay monitor, or the like. In another embodiment, the display monitor1356 is integrated into the goggle display. The goggle device 1350 sendsthe patient data to be displayed to the display device via a wiredand/or wireless communications connection.

External Display 1352: In an embodiment, the goggle controller sendspatient data to the external display (e.g., tablet computing device,personal computer, cellular phone, etc.) via a wired and/or wirelesscommunications connection. This data is compiled into one or morereports to help the patient understand the progression of the disease.The data may also be sent to an external patient portal cloud for thehealthcare provider to remotely monitor the patient's progress.

Mouse/Pad 1354: In an embodiment, the mouse/pad includes a pad with astylus pen, a plain mouse, or the like. This device 1354 is used toallow the user to provide feedback to the goggle device 1350. Themouse/pad 1354 is connected to the goggle device 1350 via a wired and/orwireless communications connection. The mouse/pad 1354 may be any inputdevice, such as a touchscreen and/or any input devices described herein,for example.

Additionally or alternatively, the eye monitoring and vision enhancementdevice 1350 is used after eye surgery to help restore vision faster andalso to help sharpen images for patients that are partially blind and/orhave clouded vision.

Conventional cellular phone apps that provide vision testing and/orAmsler grid progression testing (FIG. 21 ) lack the consistency ofdistance from the patients eye and are difficult to use. For example,users report difficulty focusing their eyes on the dot when their fingeris trying to draw a line to the distortion in conventional Amsler Gridtesting apps. Moreover, reports indicate that about 11 million Americanshave AMD and that injections can cost between $50 and $2,000. Forexample, reports claim that Medicare paid $1 billion for 143,000patients using Lucentis. Conventional techniques do not provide means oftracking how often patients really need injections. Conventionaltechniques require patients to visit the healthcare provider on aregular schedule (e.g., every 1-2 months) but there is no way oftracking progress other than what the doctor reports.

Another embodiment in accordance with aspects of the disclosure includesmonitoring a user's voice for audio irregularities that indicate, forexample, awareness, deception, intoxication (e.g., slurred speech,etc.), or the like. The speech is monitored by a smart phone, a tabletcomputing device, or a dedicated computing device as further describedherein. In an embodiment, a microphone input is operatively coupled(e.g., electrically, communicatively, etc.) to a digital-to-analogconverter (DAC). The internal microcontroller (e.g., graphics processingunit (GPU), central processing unit (CPU), digital signal processor(DSP), etc.) then performs a Fourier Transform or Fast Fourier Transform(FFT) of the incoming (e.g., real-time) or prerecorded audio sample. Thesample is analyzed for shifts in frequency over time and these shiftsare recorded to establish patterns and/or baselines. Time domainanalysis of the audio input is used to establish changes in tempo orattack based on the envelope of the signal. In addition to monitoringthe audio input, aspects of a system in accordance with an embodiment ofthe disclosure monitor video and/or other sources simultaneously orasynchronously to enhance the ability for detecting changes. Whenmonitoring video input, the entire video frame may be monitored, oralternatively, specific locations in the frame may be monitored usingmachine vision and image recognition algorithms, for example. In anembodiment, the software executing on the microcontroller uses existingapplication programming interfaces (APIs) to isolate and capture theuser's eyes by monitoring the frames and tracking the change in pixels.The filters used to isolate the user's eyes monitor the areas around theuser's eye, eyebrows, mouth, entire face, and/or entire body, forexample, for shifts and/or movement. The microcontroller may keep thecollected information (or electronic data representations thereof)locally (e.g., on a memory storage device electrically coupled thereto,etc.) or may upload it to a centralized server (e.g., communicativelycoupled to the microcontroller, etc.). The collected information may beused to compile an expert system that could be used by other users. Inanother embodiment, crowdsourced information could be used as input fora deep learning and/or machine learning system. The system is configuredto solve for (e.g., determine) various conditions including, but notlimited to, optimal drug dosing, optimal time to take/administer drugs,optimal time for certain activities (e.g., study, sleep, exercise,etc.), or the like.

Some embodiments, such ones dealing with the delivery of drugs, utilizea closed-loop system to ensure that the selected AI algorithm canproduce a correct solution. For example, a system that is monitoringbeta blockers could use blood pressure measurements to create aclosed-loop system. An alternative embodiment might make indirectmeasurements through monitoring blood pressure and skin sweat, forexample. Other embodiments may require blood or urine analysis to createa closed-loop system. In systems in which the drug amounts and/ordelivery times are configured to be modified by an algorithm, systemsand methods in accordance with the present disclosure place restraintson the system to restrict the timing and/or dosing within clinicallysafe levels.

In yet another embodiment, virtual reality is used to implement apost-operative care for a total knee arthroplasty (TKA). The care couldbe started in the hospital or after the patient has been discharged. Forexample, a typical post-operative protocol for a TKA focuses on lowerextremity exercises, which may include ankle dorsiflexion/plantarflexion, quad sets, hamstring sets, gluteal sets, short arc quad, hipabduction, heel slides, long arc quad, active knee extension/flexion,straight leg raise, and the like. Patients are encouraged to performthese exercised every 2-3 hours while awake. By creating a series gamein a virtual reality environment, the patient is guided through a seriesof actions which will require the patients to perform the requiredmovements to complete the level. The virtual reality environment mayrequire the user to specifically perform each stretch, or it may beimplemented in a way in which the patient follows a narrative and therequired activities cause the patient to stretch the required muscles.In an embodiment, the motions/stretches of the patient (e.g., legmotions, etc.) are tracked with cameras and/or position sensors, asfurther described herein. The protocol may be customized by a doctor andthe progress of the patient may be tracked (e.g., remotely via acommunications network and computing devices as described herein). If apatient does not make sufficient progress during the post-operativestretching, this could be a trigger for an alert (e.g., text message,email, report, smartphone notification, etc.) to be sent to the doctor,physical therapist, insurance agency, or the like. Although thisembodiment focused on a post-op TKA, one of ordinary skill in the artwill understand that other embodiments and exercise protocols can beimplemented for other procedures.

For example, aspects of the systems and methods described herein canprovide visual proof, such as via intraoperative pictures or videos orpost-operative imaging, to confirm that operative procedures wereperformed. For instance, an insurance carrier can use the visual proofto validate that the operative procedure was performed and performed topredefined standards, such as by analyzing the pixels of pictures,videos, and/or images. In an embodiment, the systems and methodsdescribed herein validate appropriate alignments, tissue removal, and/ortissue repair. Machine learning aspects described herein can analyzepictures, videos, and/or images to confirm. In one embodiment, themachine learning aspects automatically approve or deny operativeprocedures based on pictures, videos, and/or images that conform to ordeviate from a baseline standard. In another embodiment, the machinelearning aspects present the automatic determination to a human user tooverride or confirm the machine learning determination.

One embodiment would use pattern recognition. Another embodiment couldimplement a Deep Learning algorithm implemented with a ConvolutionalNeural Network (CNN). This type of AI requires a large number of imagesto train the system, but can be more accurate than other algorithms. Aneural network is created from a network of artificial neurons withdiscrete layers, connections and data flow. The neurons are trained toperform a task such as automatic image recognition. This can be doneusing labeled pictures of an object as an input and the output of theneuron being the label. For example during the training of the neuron,the input of an X-Ray of a “left knee” would be label as a “left knee”.The more labeled images the neurons are exposed to the more accurate thesystem becomes. In some embodiments, the system is trained with over 10million images. Each neuron gives a weighted output which is totaled togive a confidence level of the image recognition. In this example, thesystem might be 95% confident that the image is a right knee, 10%confident it is an ankle, and 1% confident it is a cat. With increasedneurons and layers and large amounts of training, the system gets betterat identifying objects. In addition to anatomy, the neurons could betrained to detect surgical implants such as total knee, unicompartmentalknee implants, intramedullary rods, dental implants. In an insuranceverification system embodiment, when an insurance claim is submitted, aDICOM image can be transmitted to the system for verification. The imageis then passed through the automatic image detection algorithm, whichfirst slices the DICOM image into “tiles” which are then passed to thefirst layer of the neural network, the output would be passed to thesecond layer and repeated for each layer until the image recognition iscomplete. The output of the system is an interpretation of what theDICOM image was of, for example “Left knee, human, total knee implant,and bone cement.” If the output of the image recognition matches thebilling code the claim would be processed automatically. If theconfidence level was for the submitted billing codes was below apredetermined threshold, the claim would be flagged for furtherinspection. If during manual inspection it is determined the submittedcodes match the image, the claim would be processed and the image couldbe used for additional training for the neurons. If the image did notmatch the billing codes the after manual inspection the claim could bedenied.

In some embodiments, both a pre-op and a post-op image are required. Inaddition to image recognition, a reverse image search of the databasecan be performed to ensure that the image submitted with the claim isnot being reused. Intra-operative pictures could be required for someprocedures instead of or in addition to X-rays.

In some embodiments, the AI techniques described herein are used fornavigation. For example, motorized wheelchairs could be converted fromnormal operation to autonomous or semi-autonomous operation. This couldbe very useful for people who have limited use of their hands of arms tonavigate the wheelchair, for instance. In one embodiment, asemi-autonomous wheelchair is equipped with sensors to look forpotential hazards or obstacles in the direction of travel. If there wasa small obstacle in the path that could be avoided, the chair couldautomatically drive around the object, then return to the original path.If the object could not be avoided, the wheelchair could be programmedto stop before hitting the object. A fully autonomous wheelchair wouldalso reroute its path to avoid obstacles. In another embodiment, it maybe desirable to have a remote transponder that is used to determine thedirection in travel. In this embodiment a small transponder is attachedto the belt or a person and the wheelchair would automatically determinea path that would follow the transponder at a predetermined distancewhile avoiding any obstacles. The chair could also avoid by stopping, orgoing around changes in the height such as stairs, curbs, or potholes.This could be done when fully autonomous or semi-autonomous. The chaircould also automate portions of navigation, such as positioning thewheelchair on a chair lift.

In addition to traditional sensors, autonomous drones could be used toprovide an additional input. For example, this could be useful incrowded areas to help determine the best path for traveling to a certainpoint. The drone can include traditional cameras and thermal cameras.The drone can communicate with the chair via Wi-Fi or other short- tomedium-range radio wave protocol. In some embodiments it might bedesirable to have more than one collaborative drone. It is alsoconsidered that a single drone could provide camera feed, or mappinginformation to more than one chair. In one embodiment, a universalprotocol could be written and used with collaborative drones such thatwhen a drone is available in an area, any equipped chair wouldautomatically connect and use drone information as an additional input.Although described herein in terms of a wheelchair, one of ordinaryskill in the art will understand that these aspects may be embodied inbicycles, tricycles, hoverboards, Segways, boats, jet skis, any type ofjet drive vehicle that can be controlled, and the like. For instance,these techniques can be linked to GPS, sonar, camera systems, light air,robotic self-taught autonomous robotic systems, and the like.

In addition, the techniques could also be embodied in an autonomousrobotic system such as a nurse, which could perform a nurse's physicalresponsibility (e.g., taking vitals, drawing blood, etc.) and/or offernursing technical support (e.g., explaining procedure, addressingquestions, obtaining consent, etc.) to the patient directly or in avirtual manner. The autonomous robotic nursing system integrated withthe AI system 104 can determine the physical and mental health of thepatient. The AI system 104 can analyze the patient information (e.g.,test results, systematic questioning, etc.) obtained from the patientand/or robotic nursing system. Once enough information is available totarget a diagnosis the AI system 104 can send the patient information tothe physician for review and/or generate a diagnosis or list of possiblediagnoses for the physician's review. If the diagnosis generated by theAI system results in a medical procedure (e.g., a procedure to treat thediagnosis), the AI system 104 may direct the autonomous robotic nursingsystem to prepare the patient by explaining the procedure, additionaloptions and the reason to provide the procedure. The AI system 104 mayalso be equipped to detect the patient's understanding of the procedureby using patient monitoring (e.g., eye movement, rate of breathing,shaking, etc.) via the robotic nursing system. In the occurrence that AIsystem 104 detects the patient is unsure of the procedure or still lackscomplete understanding the autonomous robotic nursing system wouldprovide additional information and reassurance, at the direction of theAI system.

Moreover, the concepts disclosed herein can be embodied in aself-driving or self-directed exoskeleton. A person who may have adisability weakness or wants to amply force (e.g., soldier, etc.) theexoskeleton would amplify the strength or force of an individual throughservomotors, robotic controls, hydraulic pistons, and the like. It couldbe rigidly attached to the person's frame (e.g., through pins into thebone, etc.) or it could be externally attached with form fitting cuffs,inflatable bladder type cuffs or supports, or structural supports. Itcan be built to the trunk, specific arm, leg, foot, toe, and the like.It could also be used for patients with cerebral palsy, neuromuscularweakness, strokes, CVA, and the like. This could be used forrehabilitation but also used to drive a motor, move or direct patientsto certain locations to result in essentially a self-driving exoskeletonfor those patients who have disabilities or those patients who want toamplify their forces in specific areas. This would not only increase theforce or improve range of motion, function, strength, work against someof the spasticity or rehabilitation recovery for range of motion. Itcould also be used if these are used for external to take exoskeleton toits specific location or to drive them to direct them to do a specificactivity at a specific location including lifting, moving, orexoskeleton plus self-driving autonomous vehicle controlling fine motor,specific motor, or gross motor. It could be single motor, such asservomotor. It could be a fine wheel electromagnetic motor, hydraulicpiston, or a combination thereof. It would also have superficial sensorsthat would allow tactile feedback in at the periphery or at the centraljoint which is mechanically, electrically, or electromagnetically forexample drive. Furthermore, these embodiments can be recharged. Forexample, they can be recharged via photoelectric systems, sunlight,heat, fluid motion, electromagnetics, and the like. In an embodiment,they are recharged remotely.

In yet another aspect, the artificial intelligence and/or virtualreality techniques are used for self-driving diagnostics. The generalconcept is that patients might have individual testing. They may havex-ray, ultrasound, MRI, PET scan, bloodwork, and/or echocardiogram, forexample. However, these are all individual treatments done on individualmachines at individual locations. This would also add electronicdiagnostics such as EKG, EEG, sleep labs, and the like.

When the patient comes in and has some concerns about pain, the systemsand methods can be used to initially do a scan of ultrasound, EKG, orEEG to examine if there is an abnormality identified. The system wouldthen direction the patient to the next step by using automation,artificial intelligence, and/or a possible link to a radiologist,physician, nurse, or healthcare provider who can automatically approvethe next appropriate test. For example, if a patient has pain, thesystem would direct the patient to undergo a quick scan with anultrasound. The system analyzes the data from the ultrasound to detectif there is any irregularity, such as an irregularity on a bone. Thesystem would then direction the patient to immediately have x-ray or CTto determine if there is a fracture in the bone. If the system detectsthat there is a questionable fracture, the system then automaticallyidentifies the fracture type and/or location and automatically generatea treatment program to be recommended or instituted either directly to apatient or a physician. Other irregularities that may be detected by thesystem from the ultrasound include if there is a mass or tumor or ifthere is a question of irregularity of other types of tissues.Accordingly, the system would then direct the patient to go theappropriate medical device based on the irregularity detected, such as aMM. All of these diagnostics could be in one room or one location orpotentially within one device.

Next, if there is a question and one notices increased body temperatureor sweating or increased in fluid osmolarity, a urine analysis could beperformed. The system could automatically draw blood either throughrobotic system or have a blood technician draw it, sequentially thesetests are automated and driven by this automated self-driving diagnosticsystem. If the blood is then drawn and there is a question aboutdiabetes as blood sugar might be elevated, one would scan the stomach tosee if the patient ate recently if this is a false-positive orfalse-negative and then it would go to the next system on this. If thereis question, they would do an ultrasound to scan whether the patientmight have other associated issues such as increased risks for cardiacproblems as diabetics have a small vessel disease or retinal disease.Retinal scan would go on looking at all of the diabetic potential issuesdown the algorithm. One would then look at associated diseases withdiabetes. One might check pancreas function or liver function. Thesewould go down the diagnostic list and these would all be performedbefore the physician would actually see the patients so this would beautomated diagnostics. This could also be done through a bed or othersurface so patient could lay down, especially if they are in thehospital. If they sweat excessively or move excessively, one might havesensors in the surface, EEG and EKG in the surface. If the patient movesexcessively during nighttime, while they are active, if they sweat, orif they have sensors that show body chemistry changes, automatedbloodwork would either be ordered or could be done at the bedside. EEG,EKG, or ultrasounds could also be automated either they are in a singleunit or multiple units, but they would be potentially driven straight tothe patient or a series of tests would be automatically ordered so thephysician does not have to actively get involved, but these wouldautomatically line up so that the physician would have all the tests inand already in place before they actually talk to and examine thepatient. This automated system could include a single or series oftests. It could be linked from urinalysis and bloodwork to MRI, CT, EKG,EEG, gait analysis, video monitoring so one could see specific motionpatterns.

Once the diagnosis is delivered, a treatment protocol can be given,followed by a follow-up protocol. This could be done through video,holograms, activity patterns, robotic simulation. In addition, thefollow up protocol could also be done through sequential follow-upbloodwork and follow-up testing. It is coordinated and organized atappropriate interval so the patient has expedited throughput. In oneembodiment, there could be automated testing at home so that perpatient's mobile devices and artificial intelligence the rightdiagnostics could be at someone's doorstep or someone's home eitherpermanently or temporarily, the appropriate sensors either implantableor monitored on the surface, the appropriate diagnostics such as homeultrasound diagnostics or diagnostically come to localized facility.These would all be setup in advance and multiple tests would be doneincluding ultrasound, bloodwork, and urinalysis. All reasonable andappropriate tests would be done all at either the patient's home orspecific location. It would all be automated through this mobile and/orautonomous diagnostic, autonomous therapy/treatment. Again, this wouldbe for example if someone is recovering from knee replacements automatedx-rays would be setup. These could be done at home or through a mobiledevice or with automated ultrasound, bloodwork, therapy patterns, motionpatterns, accelerometer on phone so visual aspects from the phonegoniometer, apps, or through rented or permanent diagnostics which couldbe delivered, mailed, or shipped back. These could all be linked intospecific patterns. The sensors would be linked to each other so theappropriate diagnostics could be done and then they would be returned.These would be mobile or wirelessly delivered to a central computerthrough AI. It could constantly monitor, upgrade, and also look atbehavioral modifications and behavioral treatment patterns that areknown through AR to change peoples' behavioral patterns such as if oneis a diabetic, which foods they should ignore, which activities theyshould do or which time these activities should be performed. When isthe best time to study, when is the best time to sleep, when is the besttime to perform work related activities. This could automate not justwith therapy but actual day-to-day lifting and give you best advice,best case examples, and possibly through noxious stimulants or positivereinforcements to teach you what the right activities would be. If youare a diabetic and you are eating sugars, it might give you a noxiousnot to do this or suggest alternative diet treatment. Finally, thesecould be autonomously linked to food delivery service or activitydelivered to your home so you have the appropriate nutrition,appropriate activity, or appropriate advisors. There is an entire systemand pattern for autonomous diagnostics, therapy, and/or follow-uptreatment with positive reinforcements and/or negative reinforcements.This will be acquired into one solid system.

Another embodiment could use machine vision and deep learning to createan autonomous diagnostic unit. A system could contain multiple diagnosismodalities (ultrasound, Mill, X-Ray, CT, PET, etc.) which monitors theoutput of the imaging modality using artificial intelligence to look forindicators of specific conditions or disease states, if one of theseconditions are identified during the procedure an alert could be sent tothe operator and a physician if additional imaging is required. If thephysician had previously approved the scans, or responds via a computingdevice, phone, messaging system while the patient is still in thescanning system additional diagnostic modes could be performed at thesame time. This would prevent the patient from being transported to adifferent area and requiring a second visit. In addition to imaging,other diagnostic tools could be automated as well including blood draws.Some embodiments would use robotics to perform blood draws, or theycould be done with by a phlebotomist. The phlebotomist and physiciancould receive an alert while the initial scan was occurring.

In addition to monitoring the output of diagnostic imaging system, thepatient's vitals and other sensors could be monitored. This couldinclude but not be limited to blood pressure, skin conductivity, heartrate, EKG, blood oxygen level, temperature, and results from previouslab work. These could be monitored during procedures or any time thepatient is being monitored. If the system recognizes a condition whichcould require additional blood work, an alert could be sent out tosuggest additional tests. In scenarios where a physician or insurancecarrier is required to authorize the system could trigger a notificationto the required party, then order the required test. For example, when apatient has an unexpected fever the system could notify the phlebotomistand the lab to perform a count of leukocytes.

In yet another aspect, the artificial intelligence techniques describedherein are used to remind a user to perform an activity such as when totake or apply a medication. For example, in one embodiment the AItechniques remind a parent and/or optimize when to give a medication toa baby. The AI techniques disclosed herein can incorporate a babymonitor (sensor and/or visual) and relay the information to a mobilecomputing device such as an application on a cell phone. The parent canuse the mobile computing device to input the medication prescriptioninto the application such as by scanning the prescription with themobile device. Such information inputted can include the medicationtype, dosage, and how often the medication should be administered. Theapplication then monitors the child's sleep patterns (via the babymonitor) and determines the best time to administer the medication. Forexample, if the child is sleeping, the application would not active thereminder until the child wakes up during the night—instead of waking thechild up while the child is sleeping. In particular, if the medicationis supposed to be administered every 4 to 6 hours, the application wouldwait to send a reminder to the parent to administer the medication untilthe child is awake during the interval when the medication is to beadministered—in this case between hours 4 and 6. If the child saysasleep during the entire interval, the application waits to send thereminder until the time limit is reached—in this case at 6 hours.

In another embodiment the AI techniques remind a patient and/or optimizewhen to take a medication. Biosensors that record biometric data fromthe patient are incorporated into the techniques described herein andrely that data to the application on the mobile computing device. Theapplication can then reminds and/or warns a person to take or not totake the medication based on the information from the biosensors. Forexample, if a person is trying to maximize weight loss and taking amedication to increase metabolism, the application can indicate when theperson should take the medication for maximum effect. The person caninput the medication information into the application and then theapplication would use an algorithm, the information from the biosensorsand the recommended dosage to indicate the best time of the day to takethe medication. In this case, the application may also indicate to theperson when they should or shouldn't consume certain foods based on theinformation from the biosensors and the medication information in theapplication. In one example, the medication may include caffeine, whichaffects hear rate. Some people are more sensitive to caffeine thanothers and medications that include caffeine can cause a sensitiveperson to feel light headed and dizzy. Alternatively, some people areless sensitive to caffeine and may feel zero effect from the medication.Either instance can cause the person to be less consistent in taking theweight loss medication or quit all together. The application, via thebiosensors (such as a hear rate sensor) can monitor the heart rate ofthe person immediately after taking the medication to determinesensitivity. Based on initial baseline levels, continued monitoring ofthe biometric data and/or the medications recommended dosage, theapplication can indicate how often the medication should be taken andwhen to increase the dosage.

In yet another aspect, the artificial intelligence techniques describedherein (e.g., AI system 104) can be incorporated or in communicationwith the control system of a surgical robot, such as the da Vincisurgical robotic system by Intuitive Surgical. In this embodiment, theAI system 104 can collect the data, such as surgical information orpatterns, from surgical robot and/or surgeon during medical procedures.The AI system could gather and analyze this data on a personal level(surgeon), a local level (surgical group or hospital), and/or a nationalor international level (medical procedure or type of surgical robot).The AI system 104 can use this data as an input to create a model totrain new users how to operate the surgical robot. Moreover, this datacan be combined with patient records and outcomes to generate the besttechniques/procedures regarding the surgical robot for optimal patientoutcomes and train surgeons accordingly. Similarly, the AI system 104can use the data collected to generate surgical guidance where thesurgeon would receive feedback, using the surgical robot's display,virtual reality and/or augmented reality, on the best procedure or stepsforward based on the model and/or the steps already taken by thesurgeon. In another embodiment, the AI system 104 may instruct thesurgical robot to carry out the preferred procedure, autonomously, basedon the surgical guidance model using machine vision.

For example, the AI system 104 can be used with a surgical robot toremove and/or facilitate the removal of tumor cells. Through repetitiveuse, the AI system 104 can learn which cells are tumor cells and whichcells are normal cells, allowing the AI system to operate the surgicalrobot to remove the tumor cells while leaving the healthy cells inposition, for example by neovascularization. For example, areas withhypervascular supply often are related to fast growing tumors. The AIsystem 104 can learn and/or identify which areas are more vascular anddirect the surgical robot to remove the areas that are morehypervascular and leave the areas that are less vascular or have normalvascular flow through normal vascular anatomy. Similarly, this conceptcan also apply to normal scaffold anatomy or normal cellular anatomy. Byconnecting the surgical robot to the AI system 104, the AI system canprogressively learn, as described herein, surgical robotic functionsand/or cellular functions to treat a specific area while looking forabnormalities or irregularities in function. This can be done on ageneral to a more granular basis. Moreover, the AI system's 104 learningprocess could be altered based on any sudden changes. For example, ifone removes one data point from the AI system 104, it would alter theentire stream. The data point could be removed at the end of thelearning process or the beginning of the learning process. This could bere-visited potentially for the next patient or next subgroup ofpatients. It is understood that the AI system can be used with anysurgical robot and/or surgical robot operation/technique. For example,the Ai system 104 can be used with the systems and methods describe inU.S. Pat. No. 9,155,544 and U.S. Patent Application No. 2017/0112577,the entire disclosures of which are hereby incorporated by reference.The Ai system 104 can also be used with other surgical techniques anddevices such as those disclosed in more detail in U.S. Pat. No.7,104,996 and U.S. Patent Application No. 2016/0144113, the entiredisclosures of which are hereby incorporated by reference.

In yet another aspect, the artificial intelligence techniques describedherein (e.g., AI system 104) can be incorporated or in communicationwith a graft such as grafted blood vessel in an eye, although othergrafts are within the scope of the present disclosure. The graft may becreated from a combination of a shape memory polymers and a biologic.The shape memory polymer could be constructed from a electroactivepolymers (EAP) which contracts and dilates with a change in electricalcurrent. The AI system 104 would monitor the neural impulses of thesympathetic nervous system via other patient monitors 102. These neuronsconnect with the muscle cells in the blood vessels. When these neuronssecrete norepinephrine it causes the muscles to contract. By monitoringthese impulses, the AI system 104 can correlate these impulses to otherstimuli such as stress, physical activity such as running, or otherbiological responses. The AI system 104 could also respond directly tothe neural impulses. The AI system 104 would then activate the EAP inthe graft to closely mimic the original vessels—dilating or contractingthe blood vessel. Another embodiment could use diagnostic scans(ultrasound, MRI, etc.) from multiple patients to create an AI model forthe vessel behavior. The Ai model could be preloaded and into thecontroller for the graft. This output of this model could also be usedto 3D print a graft with varying density and patterns of the EAP suchthat the behavior of the vessel behavior. These grafts could be createfrom thermally activated polymers or alloys, electrically activatedalloys, or a combination of both. The graft could also be createdwithout the biologic so that it is totally synthetic or could totallybiologic. This would not be limited to blood vessels but could be usedin any lumen including but not limited to pancreatic duct, thegastrointestinal tract, the bronchi, renal tubules, colon, etc. Thesecontractible lumens could also be in non-biologic applications such as,but not limited to, fluid flow, oil flow, gas flow, robotics, andindustrial equipment. The information gathered from the AI model couldalso be used in conjunction with diagnostic imagine to create a filterthat could compensate for the movement of the blood vessels duringobservation by predicting the movement. This could be used in endoscopy,ultrasound, MRI, and other diagnostic imaging techniques known in theart.

In one embodiment, the data shared with the AI system 104 can be minedon an individual or group basis for specific information. Moreover, inone embodiment, the AI system 104 may ask the patient to share dataand/or ask specific questions to the patient. An incentive may be givenif the patient responds. Moreover, an incentive can be given otherprofessional for information. For example, data including but notlimited to robotic surgical techniques, sleep patterns, etc. could beexchanged for data access to the input data, the output data, or couldbe a monetary benefit such as company stock or coupons. The more datathat is provided the more stock or coupons the user could earn. Thestock could be kept and the value would grow as the value of the datagrows, or they could be traded in for cash but lose out in growth.Recruiting more data miners, or recruiting more users could allow for abonus.

For example, the AI system 104 could incentivize users (e.g., patients,physicians, professionals) in order to gather information on medicalcosts. Users could fill out questionnaires and those that answer orprovide most data on a topic get bonuses or prizes or more stock. Thequestioners could be paper, electronic, or in a game format with prizesor TV show type excitement like Wheel of Fortune or Jeopardy. The dataminers go out and ask others and submit data (would need to cross checkaccuracy) but would determine what true medical costs are and collateand share. To add revenue obviously advertising but with a twist asbacked up by data ie who has best price on MM. They then would want toadvertise as the firm would have data to back up. Also, the AI system104 may incorporate a payment system, like Apple Pay, but where peopleare rewarded for their data sharing and mining. Confidentiality must beset up with encryption which protects and constantly changes but payspeople for real data rather that do it in the sly and provide a nominalvalue. Daily or hourly questions or problems sent out and those thatprovide are gaining value and can choose their return. This would makethe data mining more transparent instead of hiding the fact it ishappening. The AI system 104 would allow the information from Bluetoothenabled biometric data from wearables to be submitted to a third party,such as hospitals, surgery centers, insurance carriers, etc for variouspurposes, such as evaluation. The users could be compensated for thisinformation. Also data from smart homes, sensors, or security camerascould be collected. Individuals could approve and share some or all ofthe data. The AI system may identify and/or omit confidential dataand/or encrypt the data so that it is not shared for mining.

In another embodiment, the AI system 104 may be communication with phoneor other wearable device that is configured to actively transmit alldata or selectively transmit the data to the AI system. The user maycontrol the transmission of data thru voice activated or an on/offsystem. The AI system 104 may also “scramble” or encrypt the collecteddata to keep the data anonymous but accessible for use in data pools.This could have advantage that unlike Facebook or Amazon where the thirdparty knows where data comes from one could actively submit anonymousdata which is still valuable especially for medicine applications ordesign applications. In another embodiment, the phone or other wearabledevice “scrambles” or encrypts the data before sending the data to theAI system 104 to create anonymity while still allowing the data to bemined.

Various patient monitors (e.g., patient monitors 102), and uses thereof,for use with the AI system 104 and the AI techniques described hereinwill now be described. Generally speaking, the following patientmonitors and methods of use thereof are for collecting biometric data todetermine a condition of a patient. Such a condition can include, but isnot limited to, health information, emotional reactions, physiologicalreactions, pain, etc. Further, the patient monitors and methods of usedescribed below are generally described as standalone systems. However,it is understood the patient monitors and methods of use described belowcan be used with the AI system 104 described above. It is alsounderstood, that the AI system 104 can be in communication with thestandalone patient monitors described below and/or take the place ofsome of the components of the standalone patient monitors.

One example of biometric data that a patient monitor sensor 102 candetect is blood pressure. Typically, blood pressure is measured using asphygmomanometer. Biometrics, such as blood pressure, can provideinformation about a patient's response to stimuli such as physicalactivity and consumption of various media. In addition, various kinds ofinjuries and ailments can be associated with changes in blood pressureand blood vessels, which if detected, can be used to diagnose a patientand/or evaluate the effects of treatment. Retinal imaging and evaluationcan be used to determine a patient's blood pressure and, through variousmethods, can allow physicians to diagnose diseases and health problemsincluding diabetes, hypertension, and concussions. For example,ophthalmologists look into patients' retinas for diseases and otherproblems such as diabetes, easy bleeding, and edema on the brain.Generally, a patient's pupil is dilated during the imaging. Retinalscanning is one way to image a retina in which a small infrared lighttraces a path across the retina when the pupil is dilated. Thedifference in reflectivity of the blood vessels and the surroundingtissue allows for mapping of the retina by sensing the levels ofreflected infrared light at different points.

Referring to FIG. 24 , a retinal evaluation system (e.g., patientmonitor) for medical diagnosis is generally indicated at 1410. Theretinal evaluation system 1410 includes an imaging system 1412configured to capture highly magnified images of a subject's retina. Theimaging system 1412 is configured to transmit the images to a memory1414 for temporary or permanent storage on the memory. An imageprocessor 1416 is operatively connected to the memory 1414 to access theimage data stored on the memory. The image processor 1416 is configured(e.g., by executing processor-executable instructions stored on memory14, etc.) to process the image data and generate one or more outputsbased on the image data. For example, in one or more embodiments, theimage processor 1416 is configured to generate an enhanced rendering ERof the image for being displayed on a display 1418. As explained below,the illustrated image processor 1416 is also configured to evaluate theimage data to perform a medical diagnostic of the subject. The processor1416 can, for example, display a diagnostic output DO on the display1418. As explained further below, in certain embodiments, the imageprocessor 1416 is also configured (e.g., by executingprocessor-executable instructions stored on memory 1414, etc.) toevaluate the image data to evaluate a subject's physiological responseto a stimulus, for example, to evaluate the qualitative effect of media,to provide an input used to generate active feedback in a userexperience (e.g., a virtual reality simulation), or to perform othertypes of medical evaluations.

In the illustrated embodiment, the memory 1414, the image processor1416, and the display 1418 are all components of a laptop computer 1419(broadly, an image evaluation device) that is connected to the imagingsystem 1412 by a wireless connection (e.g., Wi-Fi, Bluetooth, etc.). Itis understood that other computing resources may be used in otherembodiments. For example, it is contemplated that a mobile device (e.g.,a smartphone or tablet computer) may be used instead of the laptop. Inaddition, images can be stored on more than one memory and/or processedby more than one processor in certain embodiments. It is understood thatthe image evaluation device (e.g., memory 1414, image processor 1416,and the display 1418) may be in communication with the AI system 104 ifthe retinal evaluation system 1410 is connected to the AI system or theAI system may take the place and perform the function of the imageevaluation device (or one or more of the components thereof).

In the illustrated embodiment, the imaging system 1412 includes asupport frame 1420 configured to be supported on the head of thesubject. An image capture device 1422 is mounted on the support frame1420 to be operatively aligned with one of the subject's retinas whenthe support frame is supported on the head of the subject. In theillustrated embodiment, the support frame 1420 is generally configuredas an eyeglasses frame. The illustrated frame 1420 includes eyepieces1424 joined by a bridge 1426 that supports the frame on the subject'snose, along with temples 1428 that extend rearward from the eyepieces tosupport the frame on the subject's ears. Although the illustratedembodiment uses an eyeglasses frame 1420 to support the image capturedevice 1422 in operative alignment with the retina of the subject, otherembodiments use other support structures to support an image capturedevice in alignment with the retina. For example, other types ofheadwear, such as goggles, a headset, a helmet, etc., may be used incertain embodiments. In other embodiments, the image capture device 1422may be mounted on a stand supported on a floor, a table, etc.

Unlike those of conventional eyeglasses, the illustrated eyepieces 1424are configured to promote dilation of the pupils blocking at least someambient light from entering the eye through the cornea. Thus, in theillustrated embodiment, the eyepieces 1424 are substantially opaquelight blocking elements positioned in front of the eyes of the subjectto block light from entering the eyes. By blocking light from eyes, theeyepieces 1424 cause the pupils to dilate. The dilated pupils provide animproved view of the retinas and thereby enhance the conditions forcapturing images of the retinas using the imaging capture device 1422.It will be understood that light blocking elements can have otherconfigurations in other embodiments. For example, in some embodiments,the imaging system can include an optical gasket that extends around theeyes and is pressed against the face of the subject to provide anoptical seal about the eyes that substantially inhibits even peripherallight from entering the eyes. Alternatively, instead of dilating thepupils, in one embodiment, the imaging system 1412 may include contactlenses that are specifically designed to allow for imaging through asmall pupil opening without dilation of the pupil. The contact lensincludes prism and magnification elements to allow for imaging of thebackside of the eye without the full dilation of the pupil. The prismelement could be temporarily engaged using electromagnetic of RFtechnology during the imaging. In this embodiment, once the imagining iscomplete, the contact lens can adjust to allow for normal vision. Inanother embodiment, contact lenses could be used to shield the eye fromvisible light to cause or facilitate the dilation of the pupils. In oneembodiment, if the eye contains a grafted blood vessel, as describedabove, the imaging system 1412 and/or AI system 104 can control thedilation and contraction of the grafted blood vessel.

In the illustrated embodiment, the image capture device 1422 comprises alight source 1430 which is configured to shine onto the eye, at whichpoint an image sensor 1432 captures an image of the retina R. Someembodiments include a digital camera as an image sensor 1432, but itwill be understood that other types of image capture devices (e.g.,including more specialized light sensors, etc.) can be used in otherembodiments. The image sensor 1432 is mounted on an eyepiece 1424 tocapture an image of the retina R as the light is transmitted to andreflects from the retina R. Detecting the quantity of light reflectedfrom different points on the retina R is sufficient to map the retina.For example, a point on a blood vessel will reflect less light thanother points on the retina. In certain embodiments, the image sensor1432 is also configured to capture images of other portions of the eye,for example the iris. The image sensor 1432 stores the captured imagesto the memory 1414 where it is accessed by the image processor 1416 forretinal evaluation as described in greater detail below.

In a preferred embodiment, the image sensor 1432 is a digital infraredcamera and the light source 1430 is an infrared light within thespectrum detected by the digital infrared camera. Because the human iseye is not sensitive to infrared light, the light source 1430 canprovide infrared light for the image sensor 1432 without constrictingthe pupil. In another embodiment, the light source 1430 creates visiblelight and the light level is controlled so that the pupil stays dilatedbut there is sufficient light for the image sensor 1432 to capture animage of the retina. In still another embodiment, the image sensor 1432and the light source 1430 operate in the visible spectrum and the imagesensor 1432 captures an image while the light source 1430 provides lightbut before the pupil constricts significantly. In other embodiments, thelight source 1430 could be a source of near infrared light, ultravioletlight, or multispectral light, with a corresponding appropriate imagesensor 1432. In one embodiment, the lights source 1430 and image sensor1432 may be part of the LiDAR system.

In one or more embodiments the imaging system 1412 comprises acontroller 1434 that is configured to automatically control the imagecapture device 1422 for capturing images of the subject's retina. InFIG. 25 , arrows connecting blocks in the diagram shown in phantomillustrate control connections between the respective components andarrows shown in solid line show the light from the light source 1430. Inthe illustrated embodiment, the controller 1434 is a dedicated componentof the imaging system 1412; in other embodiments, the controller 1434runs on a processor 1416 of the computer 1419 or another device. In someembodiments, the controller 1434 is able to modify the focal point,energy, wavelength, activation frequency, and intensity of the light orany subset of those attributes by communicating with the light source1430. The controller 1434 is operatively connected to the light source1430, and in one embodiment can control the focal point, wavelength,energy, intensity, activation frequency, etc., of the light from thelight source 1430. This permits the imaging system 1412 to be able tolook at different parts of the eye. To capture an image, the controller1434 communicates to the light source 1430 to illuminate the eye E. Theeye E reflects some of the light, and the image sensor 1432 detects thelight and generates and sends an image to the controller 1434. Thecontroller 1434 then sends the images to the memory 1414.

In some embodiments, the controller 1434 is operatively connected to aneye position sensor 1436. Preferably, the eye position sensor 1436gathers images of the eye continuously and relays the images through thecontroller 1434 to the processor 1416, which is configured (e.g., byexecuting processor-executable instructions stored on memory 1414, etc.)to determine the position of the eye and which direction the eye isfacing. The eye position sensor 1436 relays the position of the eye Eback to the controller 1434. In some embodiments, the controller 1434uses this position to evaluate how to control the direction of the lightfrom the light source. Alternatively, multiple light sources can beused, the controller at any time selecting the light source or sourcesthat will properly illuminate the eye. In at least one embodiment, thecontroller can use the position data to physically move the image sensor1432. In a preferred embodiment, the position data is not gathered froma separate sensor, but the image processor 1416 or the controller 1434determines the position based on the images from the image sensor 1432.In an alternate embodiment, multiple light sources 1430 and multipleimage sensors 1432 eliminate the need for an eye position sensor 1436.

In some embodiments, an auxiliary device 1438 also sends data to thecontroller, which is sent back to the computer 1419 to be stored in thememory 1414. The auxiliary device 1438 can be a device for collectingany data that a processor can be configured to use in conjunction withimages of the eye to make medical diagnoses. In some embodiments, morethan one auxiliary device 1438 is used. Examples of auxiliary devicesare Fitbit and similar devices, blood sugar measurement devices such asthose used by many people with diabetes, a device to estimate sweatingor hydration, and other devices to monitor blood pressure, pulse and/orirregularities in the heartbeat such as the wrist-wearable bloodpressure monitor or earpiece monitor, both presented below. Anotherexemplary auxiliary device 1438 includes an ultrasound system for an eyeand orbit ultrasound or laser imaging system for a similar evaluation isused to measure blood vessel dilatation and blood flow rate; detectforeign substances, retinal detachment, and tumors; and help diagnoseand monitor glaucoma, cataracts, and lens implants, among other uses.Other examples of auxiliary devices include thermal probes; vibratorysensors; sensors to detect electro-chemical makeup, temperature,electrical current, or color of the skin; devices to detect blood,microvascular blood, urine, or other fluid composition; sensors todetect sweating and sweat quantity, its composition, or both; devices tomeasure electrophysiological data; an ultrasound to estimate hydrationor measure other medical data; a device to measure intraocular pressure;and a thermometer. Measurements can generally be done in real time or atone or more discrete times. Other auxiliary devices are possible withoutdeparting from the scope of the disclosure.

Referring to FIG. 25A, in some embodiments a retinal scanner 1440 isused for imaging instead of the imaging system 1412. The laptop computer1419 is also shown in FIG. 25A for convenience and clarity. The retinalscanner 1440 utilizes the same controller 1434 as the imaging system1412. It includes an infrared light source 1442 that sends a low energy,narrow beam of light through an optical fiber 1444 to a light steeringdevice 1446. In an alternative embodiment, the infrared light source1442 and the light steering device 1446 are adjacent or preciselyaligned such that no optical fiber is needed between them. Thecontroller 1434 controls the light steering device 1446 so that thelight scans in a predetermined path over the retina R. An infraredsensor 1448 detects the light reflected by the retina R. The infraredsensor 1448 is preferably configured to sense a narrow range ofwavelengths and determine the intensity of the reflected infrared lightat each of many small time intervals. Preferably, these time intervalsare coordinated with changes in where the light hits the retina R. Thisinformation is sent back to the controller 1434 and stored in the memory1414. The processor 1416 determines for each time interval whether thelight hit a blood vessel (e.g., when most of the light was absorbed) oranother part of the retina (e.g., when most of the light was reflected).The processor 1416 then correlates the data and generates a retinalimage. Stored information about the path will tell what location thelight hit at each time interval and, as previously stated, at this pointthe processor 1416 has already evaluated whether the light hit a bloodvessel or not for each time interval, therefore, every point hit by thelight can be mapped as a blood vessel point or a non-blood vessel point.In other embodiments, the processor 1416 is configured (e.g., byexecuting processor-executable instructions stored on memory 1414, etc.)to recognize when the light hit the border of a blood vessel as well,among other variations that can be made without departing form the scopeof the disclosure. In some embodiments, the controller 1434 is able tomodify the focal point, energy, wavelength, activation frequency, andintensity of the light or any subset of those attributes bycommunicating with the light steering system 1556, the light source1530, or both

Referring again to FIG. 24 , in one or more embodiments, the imageprocessor is configured (e.g., by executing processor-executableinstructions stored on memory 1414, etc.) to use the images of the eyethat are stored on the memory 1414 to perform a medical evaluation. Theimage processor 1416 is configured to retrieve the captured imagesstored on the memory 1414 and to generate a map of the subject's retina(and, in some embodiments, other portions of the eye) based on thecaptured images. In certain embodiments, the memory 1414 stores one ormore baseline retina maps, such as maps of a healthy retina, maps of thesubject's retina at an earlier point in time, maps illustrative of howcertain diseases manifest in the retina, etc., which are used by theimage processor 1416 for comparative evaluation. For example, the imageprocessor 1416 can compare the retinal map based on the captured imageto the baseline to identify irregularities or changes in the retina.Suitably, when the image processor 16 detects a problem with the retina,it displays an indicative diagnostic output DO on the display 1418 orprovides another type of indication to the subject or the subject'smedical provider. In the illustrated embodiment, the image processor1416 also displays an enhanced rendering ER of the image on the display1418 so that the doctor or other medical provider can verify thediagnostic output DO based on the image. In some embodiments a baselineis displayed, such as a previous map of the subject's retina.

In some embodiments, the image processor is configured (e.g., byexecuting processor-executable instructions stored on memory 1414, etc.)to monitor blood pressure. When an image is stored in the memory, aprocessor can determine the radius of one identifiable point on a bloodvessel of the eye. Preferably, an image from a diastolic event and animage from a systolic event from about the same time are used todetermine a diastolic and a systolic pressure. In one embodiment, theprocessor 1416 refers to calibration data and interpolates betweencalibration data points, extrapolates beyond calibration points, or usesa trend line or curve based on calibration points. In some embodiments,multiple identifiable points on blood vessels are used together todetermine blood pressure. For example, the blood pressure is calculatedusing each point and the processor 1416 records the average of theresults from the different identifiable points. Other methods of using aretinal image or image of the eye to take blood pressure are possiblewithout departing from the scope of the disclosure. For example, bloodpressure may be calculated using a blood vessel radius and theelasticity of the blood vessel.

To calibrate in a preferred embodiment, multiple images, at least animage during a systolic event and an image during a diastolic event, areinput into the memory 1414 with corresponding blood pressures. The bloodpressures are measured in another way at the times the images are taken.If multiple identifiable points are selected measure, then this processis repeated for each point and they are calibrated separately. For theselected identifiable point on a blood vessel of the retina, an orderedpair consisting of a radius and the corresponding blood pressure issaved in the memory 1414 for each image. The ordered pairs are thencompiled into a table or array. When the radius is measured at theidentifiable point to take blood pressure, the processor 1416 referencesthe table from the memory 1414 and interpolates between ordered pairs,extrapolates beyond them, or uses a trend line or trend curve toestimate the blood pressure. The blood pressure is then recorded in thememory 1414.

In an embodiment, post-concussion trauma and the presence or absence ofa concussion can be monitored. If images of the eye are generatedcontinuously, then the processor 1416 can be configured (e.g., byexecuting processor-executable instructions stored in memory 1414, etc.)to monitor swelling, focus, and ability to visually follow an image oran item. Preferably, each eye will be monitored with an eye positionsensor 1436. The processor 1416 can further be configured (e.g., byexecuting processor-executable instructions stored in memory 1414, etc.)to use the images and data from the eye position sensor 1436 todetermine the eye's movements and focus. Preferably, the subject isgiven stimuli to test his ability to focus and follow with his eyes, anddata on those stimuli is provided to the processor 16. In someembodiments, a sensor is configured to sense the location of an itemthat the subject is trying to follow with his eyes. The processor 1416can then generate a model of what the reactions should have been andgive an indication of concussion trauma or lack thereof based oncomparing the results of the test to the model

Data on swelling may, in some embodiments, increase the accuracy anddetail of a concussion diagnosis. The eye position sensor 1436 or asimilar device can be configured to measure the position of the outersurface of the eye. In some embodiments, an ultrasound or a digitalcamera will be able to detect swelling by changes in the position of thesurface of the eye. Other methods of diagnosing concussion trauma arepossible without departing from the scope of the disclosure. Swellingdata can also be used for other purposes besides evaluating aconcussion.

In another aspect, a visual analog scale for pain measurement isreplaced by objective measurements. Instead of a subject simply choosinga number or using a similarly subjective alternative for painmeasurement, objective measurements such as corneal dilation can be usedto determine pain levels. This would make it easier for healthcareproviders to make decisions based on pain levels, such as dosingpainkillers, and would avoid the need to remind people of their painwhen assessing their pain level. Other measurements may be used inconjunction with corneal dilation measurements or without cornealdilation measurements without departing from the scope of thedisclosure.

Several other diseases and problems can be diagnosed in otherembodiments. The memory 1414 can be configured to store retinal imagesand other data relating to recognizing disease by evaluating the eye E.The processor is configured (e.g., by executing processor-executableinstructions stored on memory 1414, etc.) to run tests on the imagestaken of the subject's retina to determine if any of the documentedproblems might be present in the subject. Images are then compared and ahealthcare provider and/or the subject are informed of possibleproblems. Examples of diseases and problems that can be diagnosed inthis manner include diabetes, edema on the brain, easy bleeding,swelling, hypertension, vasculitis, macular degeneration, Alzheimer's,hypertension, and others. In some embodiments, a library of correlationsbetween eye evaluation data and other measured data that indicateproblems can be stored in the memory 1414. The processor is thenconfigured (e.g., by executing processor-executable instructions storedon memory 1414, etc.) to check through all potentially relevantcorrelations to find suggestions of possible problems the subject mighthave. Some diseases, such as Alzheimer's and macular degeneration, canbe detected early using an embodiment similar to one described in thisparagraph. Patterns on the retina are also thought to correlate with thelevel of activity in the subject's brain. Readings that are too high ortoo low could suggest potential problems to a healthcare providerwhether or not the device is configured to recognize the problems. Manyother diseases and health problems can be diagnosed without departingfrom the scope of the disclosure.

Referring to FIG. 26 , a stimulus response measurement system (e.g.,patient monitor) is generally indicated at 1510. The stimulus responsemeasurement system 1510 comprises an image creation system 1550 and animaging system 1512 for evaluating responses to stimuli includingresponses to the images created from the image creation system 1550.Except for the image creation system 1550, this embodiment issubstantially similar to the retinal evaluation system 1410, and similarparts are given similar reference numbers plus 100. A framework 1520,the imaging system 1512, and their possible variants are substantiallysimilar to corresponding parts in FIG. 24 , except that further detailsand variants of the imaging system 1512 are disclosed to account for theimage creation system 1550 sending light to the eye E. The imagecreation system 1550 does not have a corresponding part in FIG. 24 , andis configured to generate images to present content such as video to theuser. To control the content, a processor 1516 sends content from amemory 1514 to a controller 1534 (shown in FIG. 27A.). The processor1516 can be configured (e.g., by executing processor-executableinstructions stored on memory 1514, etc.) to request or control contentincluding sound or other stimuli in addition to video or still images.The processor 1516 evaluates images generated and stored by the imagingsystem 1512 in conjunction with information about the content anddetermines reactions to the content. As explained further below, in someembodiments the processor 1516 is configured (e.g., by executingprocessor-executable instructions stored on memory 1514, etc.) to useresults from processing images as an input to generate active, real-timefeedback to adjust stimuli to the user; this real-time adjustment could,for example, make a VR/AR game more enjoyable.

In the illustrated embodiment, the memory 1514, the processor 1516, andthe display 1518 are all components of a smartphone 1519 (broadly, aresponse processing device) that is connected to the imaging system 1512by a wireless connection (e.g., Wi-Fi, Bluetooth, etc.). It isunderstood that other computing resources could be used in otherembodiments. For example, it is contemplated that a laptop computingdevice, desktop computing device, server computing device, or anothermobile computing device (e.g., a tablet computer) could be used insteadof the smartphone. Other embodiments have an onboard computer. Someembodiments do not have a display, or have information stored on aserver that can be displayed by any of various computer displays. Inaddition, images can be stored on more than one memory and/or processedby more than one processor in certain embodiments. It is understood thatthe response processing device (e.g., memory 1514, image processor 1516,and the display 1518) may be in communication with the AI system 104 ifthe stimulus response measurement system 1510 is connected to the AIsystem or the AI system may take the place and perform the function ofthe response processing device (or one or more of the componentsthereof).

Referring to FIG. 27A, a block diagram of the stimulus responsemeasurement system 1510 is shown. It includes elements corresponding toevery element in FIG. 25A and performing approximately the samefunction, as well as an image creation system 1550. The imaging system1512 and the image creation system 1550 are depicted as sharing acontroller 1534, a computer 1519, and an eye position sensor 1536, butin other embodiments they are completely independent. Referring now alsoto FIG. 27B as well as FIG. 27A, a block diagram of the image creationsystem 1550, the phantom lines represent electrical signals while thesolid lines represent where the device transmits light, as in FIG. 25A.

Referring to FIGS. 26 through 27B, the image creation system 1550comprises a computer 1519 that sends direction to a controller 1534. Thecontroller 1534 controls a visible light source 1552 which sends lightthrough at least one optical fiber 1554 or similar device to a lightsteering system 1556, which is directed by the controller 1534 toredirect the light to form a virtual image directly on the retina. Insome embodiments, the controller 1534 is able to modify the focal point,energy, wavelength, activation frequency, and intensity of the light orany subset of those attributes by communicating with the light steeringsystem 1556, the light source 1530, or both. A similar system forcreating a virtual image on the retina is disclosed in more detail inU.S. Pat. No. 5,659,327 to Furness, III, the entire disclosure of whichis incorporated herein by reference. Optionally, the controller 1534 cancommunicate to the computer 1519 even if it is not sending informationas part of the imaging system 1512. This could be helpful fortroubleshooting, among other possible uses. For example, ifcommunications from the controller 1534 are ongoing, then the user willknow that the problem is not that the controller 1534 is out of batteryor completely broken. In some embodiments, there is no optical fiber.For example, the visible light source 1552 and the steering system 1556could be adjacent or they could be precisely aligned and have theirrelative positions rigidly fixed.

The light steering system 1556 relies on the eye position sensor 1536 todetect the position of the eye. The eye position sensor 1536 is depictedalso as being shared with the imaging system 1512, but, as noted above,can also be separate, and is still generally used in embodiments whereother data collection takes the place of the imaging system. In at leastone embodiment, the eye position sensor 1536 detects the position of theeye E and stores the position in the memory 1514, and the controller1534 is able to direct the light steering system 1556 appropriately tomove or to adjust the direction of the light based on the position ofthe eye. In other embodiments, the light steering system 156 isconfigured to receive commands from the controller 1534 that tell whereon the retina R to put each beam of light and receive the positiondirectly from the eye position sensor 1536. As in the retinal evaluationsystem 10, in some embodiments the need for the eye position sensor 1536can be eliminated by configuring the processor 1516 to calculate theposition of the eye based on data from the imaging system 1512.

While the imaging system 1512 is configured similarly to the imagingsystem 1412, the image creation system 1550 sends light to the retina Rwhich can stop pupil from dilating. Because the image creation system1550 shines visible light at R the retina, the pupils will naturallycontract, which may stop the imaging system 1512 from being able toacquire images of the retina R of sufficient quality. There are manypossible solutions to this problem. In some embodiments, the light isconsistently kept low enough to allow the pupils to dilate sufficientlyfor retinal imaging. In other embodiments, the retinal images are takenintermittently when the light is off or sufficiently low. In still otherembodiments, the imaging system 1512 is configured to acquire images ofthe pupil, cornea, and iris, and may also be configured to collectimages from the retina if possible. In at least one embodiment, theimaging system 1512 collects images from the retina of a dilated eyewhile the image creation system 1550 generates images for the other eyeof the same subject. In this embodiment, the dilated eye may requiredilating eye drops or other means of forcing dilation. Other ways ofcollecting retinal data while light is provided to at least one eye canbe employed without departing from the scope of the disclosure. In someembodiments, the light source 1530 is unnecessary because the visiblelight source 1552 provides sufficient light. In other embodiments, theretinal scanning system 1440 disclosed in FIG. 25B and the accompanyingdescription is included in place of the imaging system 1512. Inembodiments including the retinal scanning system 1440, the meansdescribed in this paragraph to sufficiently dilate the pupil are alsowithin the scope of the present disclosure.

In many embodiments, at least one auxiliary device 1538 providesadditional data beyond retinal images for the processor 1516 toevaluate. In other embodiments, the at least one auxiliary device 1538replaces the imaging system 1512. The above listing of examples ofpossible auxiliary devices in the description of FIG. 25A generallyapplies here as well. Data gathered from auxiliary devices is alsostored in the memory 1514 and the processor 1516 correlates it with thestimuli that were presented most immediately before the reactionhappened. Examples of properties to measure with an auxiliary device1538 that are particularly relevant for many embodiments includesweating, indicators of relaxing and tension of muscles, changes inposture, etc. In some embodiments, auxiliary devices 1538 that arenon-invasive are desirable, and in some, auxiliary devices 1538 that aremobile are desirable.

The processor 1516 is configured not only to generate data by evaluatingthe images collected and measurements taken, but also to correlate(e.g., by executing processor-executable instructions stored in memory1514, etc.) that data to the content provided by the image creationsystem 1560 and to evaluate responses. Consistent signs across almostall people indicate different reactions, and many of those signs arereflected in the retina. Excitement, sadness, other emotions, fear,pain, and other physiological responses can be determined by examiningimages of the retina. The memory 1514 preferably includes a database ofdifferent profiles of what retinas look like based on differentphysiological and emotional reactions, and the processor is configuredto navigate the database and determine reactions, preferably classifyingand quantifying each response found. Brain activity can also beestimated via retinal evaluation which could help to gauge interest, forexample. Other measurements taken by an auxiliary device 1538 may alsobe able to assist in objectively measuring reactions to differentstimuli. The reactions are compared with information about the stimuliat the time so that what a user was reacting to can be determined. Theprocessor 1516 then generates reports or information that people canunderstand or that the processor 1516 can use to change the stimuli ormake recommendations to a user.

Referring to FIG. 28 , an earpiece monitor (e.g., patient monitor) formeasuring blood pressure is generally indicated at 1610. The earpiecemonitor 1610 includes an earpiece 1612, and a smartphone 1619. Theearpiece 1612 includes an electronics holder 1620, a support device1622, and blood pressure sensing system 1624. The blood pressure sensingsystem 1624 may include parts that are positioned in the electronicsholder 1620. The blood pressure sensing system 1624 works generally likea sphygmomanometer, inflating a bag 1626 in the ear and sensing thechanges in sounds as the pressure in the bag 1626 matches first thesystolic then the diastolic blood pressure. The blood pressure sensingsystem stores collected data on the memory 1614 of a smartphone 1619. Aprocessor 1616 can manipulate (e.g., by executing processor-executableinstructions stored in memory 1614, etc.) the data and send it to adisplay 1618. Measuring blood pressure often using the earpiece monitor1610 allows real-time blood pressure measurements in a non-invasive way,and can be used for on-the-go, constant health monitoring.

In the illustrated embodiment, the memory 1614, the image processor1616, and the display 1618 are all components of smartphone 1619(broadly, a computer) that is connected to the earpiece 1612 by awireless connection (e.g., Wi-Fi, Bluetooth, etc.). It is understoodthat other computing resources could be used in other embodiments. Forexample, it is contemplated that another computing device (e.g., alaptop or tablet computer) could be used instead of the smartphone. Inaddition, data can be stored on more than one memory and/or processed bymore than one processor in certain embodiments. It is understood thatthe computer (e.g., memory 1614, image processor 1616, and the display1618) may be in communication with the AI system 104 if the earpiecemonitor 1610 is connected to the AI system or the AI system may take theplace and perform the function of the computer (or one or more of thecomponents thereof).

The earpiece 1612 is adapted to hold onto the ear and inflate a bag 1626inside the ear to take blood pressure. The illustrated support device1622 is configured to wrap partially around the user's ear, but couldalso, for example, be another type of support device such as glasses.Alternatively, the support device can be omitted and the earpiece 1612can be configured to hold itself in by friction against surfaces insidethe ear. The electronics holder 1620 is connected to the support device1622. A pump 1628 in the electronics holder 1620 is configured toinflate the bag 1626. The bag 1626 is configured to push against bloodvessels in the ear when inflated. A sound sensor 1630 is attached to thebag 1626 and is preferably pushed against the skin near a blood vesselwhen the bag inflates. Other embodiments are possible, for example, thetail portion can be replaced by another type of support device 1622,such as glasses, or the support device 1622 can be omitted and theearpiece 1620 can be configured to hold itself in by friction againstsurfaces inside the ear.

The pressure sensing system 1624 works by the same principles as asphygmomanometer, in an embodiment. The bag 1626 is inflated to apressure above the blood pressure and slowly deflated. The bag 1626 hasan internal pressure sensor 1632 that senses the pressure in the bag andcommunicates the pressure to the controller 1634. When the bag pressureis equal to the systolic pressure, turbulent blood flow generates adifferent sound. The sound sensor 1630 is sufficiently sensitive to pickup the change. When the bag pressure is lowered to the diastolicpressure, the blood flow becomes laminar and the noise returns tonormal.

Referring to FIG. 29 , a block diagram of the earpiece monitor 1610, thecontroller 1634 is directed by the smartphone 1619. When the controllerreceives a command to measure blood pressure, it tells the pump 1628 toinflate the bag 1626. As the bag inflates, the pressure sensor 1632internal to the bag 1632 communicates its measurements back to thecontroller. Once a previously determined pressure is reached, thecontroller 1634 stops the pump 1628 and communicates to the pump 1628 toslowly deflate the bag 1626. As the bag 1626 deflates, the sound sensor1630 detects changing sounds from the blood vessels for calculatingheart rate and is configured to send the data to the controller 1634.The controller 1634 sends data collected to the smartphone 1619. Inother embodiments, the data is sent to a server or another type ofcomputer such as a tablet, laptop, server, or desktop. An optionalbutton 1636 is included in some embodiments to turn the earpiece 1612 onand off. In the alternative, some embodiments have earpieces that areturned on and off remotely, for example by a signal from the smartphone1619.

In other embodiments, instead of an inflatable bag 1626, the earpiece1612 positions a Doppler ultrasound imaging system (FIG. 30B) or a laserimaging system (e.g., LiDar) at a point near blood vessels of the ear.The radius of at least one identifiable point on a blood vessel ismeasured, and compared to calibration data to determine the bloodpressure. Preferably, the earpiece monitor 1610 is calibrated in such anembodiment. To calibrate the device, simply take blood pressure whilethe device is measuring the radius at the identifiable point and thencorrelate and store the collected pressures and radii as ordered pairsor as an array. While the accuracy will increase if calibration happensat multiple blood pressures, calibrating once gives a systolic anddiastolic pressure. The processor 1616 can be configured to interpolateor extrapolate estimated blood pressures from calibration data.

In other embodiments using a Doppler ultrasound imaging system or alaser imaging system, the flow rate is tracked to determine change inblood pressure. The change in pressure can be determined using thefollowing equation:Flow=Δpressure RWhere: R=8LηπR4

-   -   r=radius inside the vessel    -   L=vessel length    -   η=blood viscosity

If the earpiece monitor 1610 is then calibrated to the wearer usinganother method of determining blood pressure, such as a sphygmomanometerintegral to a system for calibrating the earpiece monitor 1610, then theblood pressure can be estimated from the calculated change in pressure.Preferably the device is calibrated at multiple pressures.

Using the above relative pressure measurement, aspects of the presentdisclosure calibrate the readings with a traditional cuff to create asystem that is enabled to track changes in blood pressure for remotemonitoring. As illustrated in FIG. 30A, an exemplary system includes atraditional cuff that includes a controller, a disconnecting wire, and awrist monitor that includes a transducer. In one embodiment, the systemis wireless and does not include the disconnecting wire. When the wristmonitor is an ultrasound monitoring system, the system is placed overthe user's wrist and the relative pressure is measured using the aboveequations. To calibrate the system, the cuff with integrated pumpcircuitry is attached to the system. The controller of the cuff thencalculates the conventional systolic and diastolic pressures. Thesemeasures are used as a baseline measurement and increases and/ordecreases from the baseline measurements are calculated based off rangesin blood flow.

Referring to FIG. 30B, a wrist-wearable monitor (e.g., patient monitor)for blood pressure is generally indicated at 1710. The wrist-wearablemonitor 1710 comprises a smartphone 1719 and a wristpiece 1720. Theembodiment shown uses a Doppler ultrasound imaging system 1722 tomeasure the radius of at least one known blood vessel. The radius isstored in a memory 1714 and a processor 1716 can use the radius toestimate blood pressure. By timing the changes in radius of the bloodvessel, heart rate can be measured as well. A display 1724 is configuredto optionally display blood pressure and heartbeat, and a controllercontrols the device and communicates with a smartphone 1719.

In the illustrated embodiment, the memory 1714, the processor 1716, anda display screen 1718 are all components of the smartphone 1719 that isconnected to the wristpiece 1720 by a wireless connection (e.g., Wi-Fi,Bluetooth, etc.). It is understood that other computing resources couldbe used in other embodiments. For example, it is contemplated thatanother computer (e.g., a laptop, sever, desktop, or tablet computer)could be used instead of the smartphone. In addition, images can bestored on more than one memory and/or processed by more than oneprocessor in certain embodiments. It is understood that the smartphone1719 (e.g., memory 1714, image processor 1716, and the display 1718) maybe in communication with the AI system 104 if the wrist-wearable monitor1710 is connected to the AI system or the AI system may take the placeand perform the function of the smartphone (or one or more of thecomponents thereof).

Referring to FIG. 31 , the body 1726 of the wristpiece 1720 comprises aflexible wristband that is preferably worn slightly tight in order toget the best reading from the Doppler ultrasound imaging system 1722.The body 1726 partially encloses a Doppler ultrasound imaging system1722, and holds a controller 1734, the display 1724, and buttons 1726 inplace. In alternate embodiments, the wristband 1712 is configured like awatch with a clasp or buckle.

A wristpiece 1720 is preferably configured to position the Dopplerultrasound imaging system 1722 over the soft underside of a forearm of auser, near the user's wrist. The Doppler ultrasound imaging system 1722measures the radius of a known blood vessel and sends that data to thecontroller 1734. The controller 1734 sends data to the display 1724 andto memory 1714 the smartphone 1719. In some embodiments, no display isused in the wrist-wearable monitor, and in some other embodiments, thedata is not transmitted and the data is accessed through the display1724. Buttons 1726 are near the display 1724 and allow the display 1724to be turned on and off for privacy and allow the device to be turnedoff to conserve battery. In some embodiments, the wristpiece 1720 mighthave minimal internal circuitry and transmit the raw measured data tothe memory 1714, while in other embodiments the controller can calculateblood pressure, average blood pressure, etc. The wristpiece 1720preferably has internal memory as well. In some embodiments, the datacollected it could be transmitted to a server and made available to theuser's physician, via a smartphone application, via a website, or both.The data could also be sent to an insurance carrier or other interestedthird party. Some embodiments include a laser imaging system (not shownin the drawing) instead of a Doppler ultrasound imaging system 1722. Alaser imaging system can collect the same information in a similarmanner and relay the information in the same way as the Dopplerultrasound imaging system described herein.

In a preferred embodiment, previous calibration allows the processor toaccurately estimate blood pressure based on radius measurements storedin the memory 1714. Calibration can be done according to the methoddescribed above in the description of FIGS. 28-30A. 30A. After thedevice is calibrated, the processor can interpolate between recordedpoints or extrapolate beyond them. If blood pressure is measured severaltimes for calibration, then a trend line or a trend curve could begenerated to account for noise.

In another embodiment, a Doppler ultrasound imaging system 1722 or alaser imaging system is used to track change in blood pressure over adistance in a blood vessel and estimate blood pressure. The flow rateand the radius can be measured, and the change in pressure can bedetermined using the equations disclosed in relation to calculatingblood pressure via ultrasound or laser imaging system in the earpiecemonitor 1610 of FIGS. 28 and 29 . In an embodiment, the wrist-wearablemonitor 1710 is then calibrated to the wearer just as described above,except that changes in pressure rather than radii are correlated withpressures. As described above, the processor 1716 can then useinterpolation, extrapolation, or a trend line or trend curve tocalculate blood pressures. Other methods are also possible withoutdeparting from the scope of the disclosure.

A technical advantage of using a wrist-wearable monitor 1710 instead ofan earpiece monitor 1610 is the availability of large blood vessels. Awrist-wearable monitor 1710 will generally lead to more preciseestimates of blood pressure than an earpiece because a given measurementerror of, for example, 0.1 millimeters, would be a much largerpercentage error when measuring the diameter of a small blood vessel inthe ear than when measuring the diameter of a comparatively large bloodvessel in the wrist.

Other embodiments include a wristpiece 1720 similar to the one shown inFIG. 31 that precisely detects the radius of at least one blood vessel.Using the elasticity of a known blood vessel, a known radius of the samevessel, and a blood pressure corresponding to the known radius, thecurrent blood pressure can easily be calculated. To estimate elasticity,the device can be calibrated to the user. Elasticity can be estimatedusing calibration data, where each point of calibration data preferablyincludes the radius of the known blood vessel and a corresponding bloodpressure. Other methods of estimating elasticity are also possible

Both the wrist-wearable monitor 1710 and the earpiece 1610 are not onlystandalone devices, but also, in some embodiments, part of a device thatincorporates retinal imaging, imaging of other parts of the eye, orboth. Performing retinal imaging in real time and using one of thedisclosed devices to determine blood pressure in real time, with orwithout other measured input data, provides a wealth of informationabout physiological and emotional responses as well as health. Apreferred embodiment includes the retinal evaluation system 1410 or thestimulus response measurement system 1510 in combination with either theearpiece 1612 of the earpiece monitor 1610, or the wristpiece 1720 ofthe wrist-wearable monitor 1710.

Any of the disclosed aspects can be configured so that medical data canbe collected by insurance companies. This is especially useful forembodiments that are worn during everyday activities and thatcontinuously monitor health, because it allows insurance companies toengage in price discrimination, encourage low-risk behavior, and suggestdoctor's visits when the data shows that it will statistically save theinsurance company money. Insurance companies can use the data to setrates or offer discounts. To encourage healthy behavior, discounts canbe offered based on specific health outcomes that are more within thecontrol of the insured than medical outcomes that directly impact thecompany's bottom line (i.e., the blood pressure of the insured is morewithin the control of the insured than whether the insured has a heartattack). Other forms of price discrimination are also enabled. Theinsurance company can also use the data if they question the need fordifferent medical procedures and to suggest to customers at high riskfor different problems that they see a doctor about those problems.

Having now described various physical aspects, in addition to somebriefly described methods, more general methods of using devices similarto those presented above are disclosed. It is understood the methodsdisclosed below can be performed by the devices alone or with the AIsystem 104 or a combination of both.

Referring to FIG. 32 , an exemplary method of workout tracking andmeasurement is generally indicated at 1900. In an aspect, method 1900 isan algorithm comprising processor-executable instructions embodied on astorage memory device of a computing device to provide workout trackingvia a software environment. For example, the algorithm of method 1900may be provided as processor-executable instructions that comprise aprocedure, a function, a routine, a method, and/or a subprogram utilizedindependently or in conjunction with additional aspects of the methods,systems, and/or devices described herein according to exemplaryembodiment of the disclosure. For example, method 1900 is executed by acomputer, such as one or more of computer 1419, computer 1519, computer1619, and computer 1719 in accordance with one or more embodiments ofthe disclosure and/or with the AI system 104.

First, the device is turned on in step 1902. Initial measurements aretaken in step 1904 to use as a resting baseline for the day. Based ongeneral guidelines and known information about the user, safetyparameters are set for each input in step 1906. In addition to data fromretinal imaging as further described herein, an auxiliary device 1438can be used to collect data. Any of the listed auxiliary devicepossibilities that are mobile and non-invasive are helpful in someembodiments as auxiliary devices, as is a GPS, among otherpossibilities. For all medical inputs measured, a safe range isdetermined or referenced. At decision point 1908, if any input is out ofrange, the user is notified immediately in step 1910, and other stepsmay stop in some embodiments. This check is preferably made with everymedical measurement. In some embodiments, a retinal evaluationdetermines if the person is in pain, and if the person is in enough painto pose a danger, then in step 1910 the device will tell them to stop ordecrease the intensity of their exercise. While inputs are withindetermined safe ranges at 1908, the process proceeds to 1912. At step1912, the user decides if she wants a goal for the workout. In someembodiments, the device optionally calculates a goal based on the timethe user plans to spend and the longer term goals of the user. If theuser inputs a desire to have a goal at 1912, the goal or goals arecalculated or input at step 1914 and the workout can begin. Steps 1916to 1920 happen continuously during the workout. All input measurementsare recorded in step 1916 and compared to the daily goal in step 1918.In step 1920, the visual display displays data to the user, includingdata indicative of whether she is on track to meet her goal or goals.When the workout is over, summary data and how it compares to theworkout goal or goals is displayed on the visual display in step 1922.Examples of goals that some embodiments could track include distancebiked, estimated calories burned, speed run, peak heart rate, and thelike. In one embodiment, the summary data includes comparisons, such aspercentile rankings, to other users of similar ages and physical makeup(e.g., weight) which may be compiled by the AI system 104.

If at step 1912 the user does not choose to use the device to track anygoals, then the device records all inputs in step 1924. Data relating tothe inputs is displayed in step 1926. In step 1924 and 1926 in someembodiments, a processor uses retinal evaluation to estimate energylevels and determine if the device should encourage the user to increasethe intensity, or suggest that the current intensity level might not besustainable for the time the user plans to exercise. In step 1928,summary data is displayed on the visual display when the workout ends.The data displayed during and after the workout can be customizable insome embodiments. Any input or calculated figure such as speed,averages, current numbers, or other statistically calculated numbers canbe displayed, preferably according to the user's preference. Especiallyin the case where goals are not displayed, personal highs, personalaverages, or other baselines can be displayed for comparison. Preferablythe device would communicate in real-time through a smartphoneapplication or a wearable display, but there are other possibilitiesincluding projecting an image onto the retina.

Regardless of whether the goal-tracking features were used, in step 1930the device takes data from multiple workouts and recommends changes ifit can support them by data collected, including any previously storeddata. The data is stored in step 1932 to provide long-term access to theuser. In some embodiments, this data is correlated with other healthdata which could include any of the data already mentioned in otherparts of the specification.

Referring to FIG. 33 a method of collecting data from a focus group,generally indicated at 2000, is disclosed. In an aspect, method 2000 isan algorithm comprising processor-executable instructions embodied on astorage memory device of a computing device to provide collection ofdata from a focus group via a software environment. For example, thealgorithm of method 2000 may be provided as processor-executableinstructions that comprise a procedure, a function, a routine, a method,and/or a subprogram utilized independently or in conjunction withadditional aspects of the methods, systems, and/or devices describedherein according to exemplary embodiment of the disclosure. For example,method 2000 is executed by a computer, such as one or more of computer1419, computer 1519, computer 1619, and computer 1719 in accordance withone or more embodiments of the disclosure and/or with AI system 104.

In step 2001, content to evaluate is identified. Content could includetrailers for movies, music, political or other ads, books, andentertainment content such as games or movies. It could also includeother videos, sound recordings, or something live. In step 2002, a focusgroup is assembled. This can be done physically or virtually. Forexample, anyone with the right retinal imaging system can participatefrom anywhere. In some instances, people might be selected without theirknowledge. Bases for selection in that case would most likely include aprior more general consent and choosing the content. In step 2003, thecontent is presented and data is gathered regarding reactions. The datacould be retinal images, but could also include changes in bloodpressure and heart rate; other data that indicates the reaction to thecontent such as sweating, coughing, fidgeting, or relaxing, for example;or a combination of multiple pieces of raw data. Preferably, the data isgathered automatically. As data is gathered or at a later point, it isanalyzed in step 2004. Not only can the people running studies or focusgroups get quantified reactions in general, but they can see whatreactions were like at different times, and thereby understand morespecifically what people were reacting to. Step 2005, which may be doneconcurrently with, before, or after step 2004, increases the precisionof the results. In step 2005, all reactions are correlated with existingdemographic information for each participant. In some cases otherrelevant information is asked of participants and is also correlated.For example, if a focus group is evaluating a trailer for a new StarWars movie, then knowing who was already a Star Wars fan is important.Step 2005 is omitted in some embodiments. A report is prepared in step2006 to help the study's sponsor understand things like what reactionsthe content evoked, what parts were most popular, how differentdemographic groups responded, etc. Finally in step 2007 the sponsor usesthe information as she sees fit. For example, the content can beimproved, targeted to the right audience, or discarded. It could also bemade in different versions, to target different groups, for example.

Method 2000 has broad application to market, sociological, political,and psychological studies. Evaluating images of the retina and otherparts of the eye can reveal excitement, sadness, other emotionalresponses, non-emotional responses such as pain, and physiologicalresponses. This would be useful for getting an objective measure of thereactions of focus group members looking at new products or ads,allowing physicians or psychologists to gauge appropriateness ofreactions to stimuli including pain, seeing what people think ofpolitical ads, determining whether people like something that they won'tadmit they like, etc. In some embodiments, a person or processorcontrolling the stimuli could receive reports of reactions in real timeand adjust stimuli to select content for a subject or group based onreactions to the previous content.

Methods similar to method 2000 can be used to track individual responsesfor use by psychologists, doctors, or simply for curious users. Forexample, a psychologist can see whether someone's reactions are normaland what they tell about a person based on data that the person mightnever be able to communicate. A physician can do more than checkingreflexes in a traditional way, and see if the retina or othermeasurements indicate that a patient has healthy internal reactions todifferent stimuli.

Some embodiments appropriate for method 2000 are mobile and a user couldwear one or more devices implementing method 2000 all throughout anormal day, while others not wearable and in some cases requirecomparatively large machines to take the desired measurements. Forexample, in some embodiments, the retina scanner and/or monitor can beintegrated into a smartphone, tablet and/or TV to monitor the user'sresponse to the screen.

Referring to FIG. 34 , some embodiments could be used for recommendingcontent and selecting ads to show. If an embodiment such as theembodiment illustrated in FIG. 3 is used for significant contentconsumption such as videos, then some embodiments could go even furtherthrough continual monitoring via retinal imaging and evaluation. Thedata could be used to determine whether a user wants to skip the introof his favorite show, estimate how much content the user wants to havesuggested, and estimate when ads can be shown for maximum attention orfor minimum annoyance by tracking the user's reaction as he sees ads atdifferent times. It would also aid in ad selection by watching hisreaction as he sees the ads. For example, if he gets excited often bytrucks, then he would see more trucks, and which trucks excited himwould be tracked to allow more targeted advertising. Recording suchreactions could also be used for advertising via other media includingphone, email, or traditional mail.

FIG. 34 discloses a method, generally indicated at 2100, that can beused for recommending content and selecting advertisements. In anaspect, method 2100 is an algorithm comprising processor-executableinstructions embodied on a storage memory device of a computing deviceto provide content recommendation and/or advertisement selection via asoftware environment. For example, the algorithm of method 2100 may beprovided as processor-executable instructions that comprise a procedure,a function, a routine, a method, and/or a subprogram utilizedindependently or in conjunction with additional aspects of the methods,systems, and/or devices described herein according to exemplaryembodiment of the disclosure. For example, method 2100 is executed by acomputer, such as one or more of computer 1419, computer 1519, computer1619, and computer 1719 in accordance with one or more embodiments ofthe disclosure and/or with AI system 104.

As the device (e.g., computer) performs step 2102, playing user selectedcontent, it continuously or intermittently performs step 2104, imagingthe eye and evaluating the images for reactions to content. Step 2104also includes gathering images from any applicable auxiliary devices.The timing of step 2104 can be determined by analysis of the content.Things identified in the content that might trigger a reaction, or“Reaction Triggers,” can be used to know when to acquire an image of theretina. In step 2106, for each reaction to the content that is captured,the time of the reaction is used to determine which Reaction Triggersthe user might be reacting to. In this embodiment, a model called theuser-likes model uses “Like Factors,” which are assigned to ReactionTriggers to determine what a user is likely to enjoy. One thing can haveseveral Reaction Triggers ranging from concrete to abstract (e.g.,swordplay to violence) and from specific to general (e.g., from “redMaserati” to “car”). At decision point 2108, each Reaction Trigger ischecked to determine if there is a Like Factor for it. If not, then instep 2110 a quick estimate is made of how much the user liked it and aLike Factor is generated in step 2112. If sufficient data is availableand the program is able to use it, then a more specific classificationof the reaction is recorded. For example, if the user finds it funnyinstead of thrilling, that information may help understand what userswant. Understanding whether the user is laughing, on the edge of hisseat, or fascinated can help sort through which Reaction Triggers aredriving his reaction as well if several happen at about the same time.Additionally, preexisting Like Factors can be consulted in generatinglike factors. For example, if a user loves dogs, but no Like Factorexists for a collie, and a reaction trigger that has a negative LikeFactor happens at the same time as the user sees a collie, then a goodreaction might be attributed entirely to the collie.

If a Like Factor already exists for the Reaction Trigger, then theprogram assesses at decision point 2114 whether the reaction generallyfit the expectations of the user-likes model. If it did, then theappropriate variable is recorded in step 2116 to indicate that only veryminor adjustments to the user likes model need to be made. If theuser-likes model was not close, then the variable mentioned is set tomake a slightly larger adjustment to the model in step 2118. After thevariable for how much to change the model is set, the Like Factor foreach Reaction Trigger is increased or decreased appropriately in step2120. At this point, the model is fully updated with the reactioninformation. At decision point 2122, the program determines if it shouldlook for ads to show. In similar methods, the answer may always be yesor always be no. If yes, then in step 2124 ads are sought usinginformation from the user-likes model. After finding potentiallyrelevant ads, the program determines which ads to display in step 2126.This decision could optionally weigh information about which ads werefound relevant in the past. After ads are selected or if ads are notsought, content that might be recommended is searched in step 2128 basedon data from the user-likes model. Finally, in step 2130, contentsuggestions are selected and displayed. When the user chooses suggestedor other content, the process begins again.

Steps 2124 and 2126, searching for and selecting ads, can also beperformed separately and can be used to advertise in other ways. Forexample, more information could be sent by phone, email, or traditionalmail. Once a user has spent enough time with the device, commercials aremore relevant than software of similar sophistication to that used onthe device has been able to make them in the past. Further, similarlyrelevant ads via other media can be presented to the user.

Referring to FIG. 35 , a method of using a retinal evaluation device forgaming simulations is generally indicated at 2200. In an aspect, method2200 is an algorithm comprising processor-executable instructionsembodied on a storage memory device of a computing device to provideretinal evaluation for gaming simulations via a software environment.For example, the algorithm of method 2200 may be provided asprocessor-executable instructions that comprise a procedure, a function,a routine, a method, and/or a subprogram utilized independently or inconjunction with additional aspects of the methods, systems, and/ordevices described herein according to exemplary embodiment of thedisclosure. For example, method 2200 is executed by a computer, such asone or more of computer 1419, computer 1519, computer 1619, and computer1719 in accordance with one or more embodiments of the disclosure and/orwith AI system 104.

In embodiments designed for entertainment or leisure, feedback gleanedfrom retinal imaging and evaluation can help a computer determine what auser enjoys. A game could use this information to determine pace,intensity, and content provided to the user, for example. In step 2202,the user starts the game. The gameplay proceeds based on a model of whatthe user likes in step 2204. If the user is new to the game, a defaultmodel can be used, or the game can be adapted to work without a model aswell. All other steps preferably occur while step 2204 continues. As theuser plays, the retina is imaged at step 2206. In addition to or insteadof imaging the retina, other measurements could be taken that indicatereaction, such as heart rate and blood pressure for example. At decisionpoint 2208, the user's enjoyment level is calculated by evaluating theretinal image. If the enjoyment level is sufficiently high, or perfect,the game continues without any updates to the model, and retinal imagesare captured and evaluated in step 2206 until an adjustment to the gameis needed. If the enjoyment level is high, but could be higher, thenonly a slight adjustment is needed. There is a slight delay 2209 and instep 2210 a variable is adjusted to indicate that updates to the modelshould be smaller than they would be. At this point, or immediatelyafter enjoyment level is found to be poor, a reason for lower enjoymentis hypothesized as shown at decision diamond 2212, which determinesupdates to the model of the user's preferences. For example, if the userseems bored, then based on previous experiences and how well the user isdoing, the game might determine that it is too easy or slow, andincrease pace or intensity or both as indicated at step 2214. In thealternative, it might decide that retinal images indicate boredom, butthat the player is bored of the style, and adjust the style of gameplayin step 2218. On the other hand, if the user is frustrated, then theintensity and/or pace can be lessened at step 2216. In any of steps2214-2218, or in other adjustments to the model of which these are onlyexamples, the gameplay is not adjusted directly, but the model isupdated, which will determine changes in gameplay. In other embodiments,the game could also be directly updated so that the model is stable andthe difference can still be felt quickly. Gameplay is continued based onthe updated model, returning to step 2204.

Many features of different embodiments such as those described in theprevious paragraph could be done with opaque eyepieces, which can beused for a virtual reality, or alternatively with darkened buttransparent lenses, which can be used, for example, for an augmentedreality simulation. Information collected to determine pace, intensity,etc. can be used not only to determine what the player wants at thetime, but can be used more generally for the player's experience withthe game and in some cases even to evaluate potential updates for allplayers.

Method 2200 is used for training simulations in one or more embodiments.For example, in a high intensity military or self-defense training, theretinal images could reveal data relating to whether the person trainingfeels scared. If the person does not feel scared, perhaps somethingcould change to increase the intensity of the simulation in an attemptto force the person training to practice reacting to fear.

FIG. 36 is a diagram of a method of use for tracking health, which isgenerally indicated at 2300. In an aspect, method 2300 is an algorithmcomprising processor-executable instructions embodied on a storagememory device of a computing device to provide health tracking via asoftware environment. For example, the algorithm of method 2300 may beprovided as processor-executable instructions that comprise a procedure,a function, a routine, a method, and/or a subprogram utilizedindependently or in conjunction with additional aspects of the methods,systems, and/or devices described herein according to exemplaryembodiment of the disclosure. For example, method 2300 is executed by acomputer, such as one or more of computer 1419, computer 1519, computer1619, and computer 1719 in accordance with one or more embodiments ofthe disclosure and/or with AI system 104.

In step 2302 medical data is gathered such as heart rate, informationfrom retinal evaluations, blood pressure, and insulin and blood sugarlevels, among other possibilities. Habits relating to health are alsomeasured or entered by the user. For example, devices to measure sleepare known in the prior art. The user could record meals and snacks, aswell as exercise. The data is analyzed in step 2304 to look for risksand suggestions for improvement. Comparisons with general norms andstatistics of the user's demographic might be made. In some embodiments,the suggestions estimate quantified rewards for changing habits. Forexample, if the device has data that shows that the user generally hasmore energy for 50% of the day when they go to bed 30 minutes earlier,or burns 500 more calories throughout the afternoon if they work out inthe morning, then the device can show the reward they get for one ofthose healthy behaviors. Tables of known data are preferably beconsulted to ensure that promised rewards are reasonable.

Generally, when the device determines that there are no serious risks atdecision point 2306 then the information is stored in a user-accessibledatabase in step 2308. It could also be accessible to the user's doctor,insurance provider, both, or another trusted third party. Periodicallyor on request, the device suggests ways the user can improve health instep 2310. As discussed, estimated results from changes might bepresented. In some embodiments, a wearer can use it to keep hisphysician apprised so that his physician knows if he has been compliantwith what the physician has asked. However, when the device determinesthere is a serious and immediate health risk at 2306, then in step 2312the device will immediately notify the user as well as predeterminedthird parties such as the user's doctor. In step 2314 the user's doctorand the user get an alert (e.g., a text message, an email, etc.) with asummary of the risk and a link to more information.

To help with overall health, sleep could also be tracked by anembodiment configured for a method similar to method 2300. Using ways totrack sleep that are known in the art, such an embodiment could, forexample, correlate sleep patterns with objective indicators, e.g., fromretinal images, subjective reports, or both of how well the user isfeeling. It could then estimate when and for how long the user shouldsleep and how dramatically it will affect the user's mood, health, andenergy. Anything else that would regularly affect mood, health, andenergy such as exercise, eating, and drinking, could be recorded,measured, or both. In some embodiments, Fitbit or similarhealth-tracking devices could be incorporated.

In any of the embodiments disclosed, tracking and analyzing habits andhealth indicators could not only benefit the user, but could also helpadvance medical science if shared.

Aspects of the systems and methods described herein are furtherdescribed in U.S. Pat. Nos. 7,182,738 and 9,474,847, the entiredisclosures of which are expressly incorporated herein by reference,including the contents and teachings of any references containedtherein.

Embodiments described herein include the communication of data and/orsignals among various components that are electrically and/orcommunicatively coupled. In an exemplary and non-limiting embodiment,the electrical and/or communicative couplings described herein areachieved via one or more communications networks capable of facilitatingthe exchange of data among various components the systems and/or devicesdescribed herein. For example, the one or more communications networksmay include a wide area network (WAN) that is connectable to othertelecommunications networks, including other WANs or portions of theInternet or an intranet, including local area networks (LANs) and/orpersonal area networks (PANs). The one or more communications networksmay be any telecommunications network that facilitates the exchange ofdata, such as those that operate according to the IEEE 802.3 (e.g.,Ethernet), the IEEE 802.11 (e.g., Wi-Fi™), and/or the IEEE 802.15 (e.g.,Bluetooth®) protocols, for example. In another embodiment, the one ormore communications networks are any medium that allows data to bephysically transferred through serial or parallel communication channels(e.g., copper wire, optical fiber, computer bus, wireless communicationchannel, etc.).

In addition to the embodiments described above, embodiments of thepresent disclosure may comprise a special purpose computer including avariety of computer hardware, as described in greater detail below.

Embodiments within the scope of the present disclosure also includecomputer-readable media for carrying or having computer-executableinstructions or data structures stored thereon. Such computer-readablemedia can be any available media that can be accessed by a specialpurpose computer. By way of example, and not limitation,computer-readable storage media include both volatile and nonvolatile,removable and non-removable media implemented in any method ortechnology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media are non-transitory and include, but are not limited to,random access memory (RAM), read only memory (ROM), electricallyerasable programmable ROM (EEPROM), compact disk ROM (CD-ROM), digitalversatile disks (DVD), or other optical disk storage, solid state drives(SSDs), magnetic cassettes, magnetic tape, magnetic disk storage, orother magnetic storage devices, or any other medium that can be used tocarry or store desired program code means in the form ofcomputer-executable instructions or data structures and that can beaccessed by a general purpose or special purpose computer. Wheninformation is transferred or provided over a network or anothercommunications connection (either hardwired, wireless, or a combinationof hardwired or wireless) to a computer, the computer properly views theconnection as a computer-readable medium. Thus, any such connection isproperly termed a computer-readable medium. Combinations of the aboveshould also be included within the scope of computer-readable media.Computer-executable instructions comprise, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions.

The following discussion is intended to provide a brief, generaldescription of a suitable computing environment in which aspects of thedisclosure may be implemented. Although not required, aspects of thedisclosure will be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by computers in network environments. Generally, programmodules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Computer-executable instructions, associated datastructures, and program modules represent examples of the program codemeans for executing steps of the methods disclosed herein. Theparticular sequence of such executable instructions or associated datastructures represent examples of corresponding acts for implementing thefunctions described in such steps.

Those skilled in the art will appreciate that aspects of the disclosuremay be practiced in network computing environments with many types ofcomputer system configurations, including personal computers, hand-helddevices, multi-processor systems, microprocessor-based or programmableconsumer electronics, network PCs, minicomputers, mainframe computers,and the like. Aspects of the disclosure may also be practiced indistributed computing environments where tasks are performed by localand remote processing devices that are linked (either by hardwiredlinks, wireless links, or by a combination of hardwired or wirelesslinks) through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

An exemplary system for implementing aspects of the disclosure includesa special purpose computing device in the form of a conventionalcomputer, including a processing unit, a system memory, and a system busthat couples various system components including the system memory tothe processing unit. The system bus may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Thesystem memory includes nonvolatile and volatile memory types. A basicinput/output system (BIOS), containing the basic routines that helptransfer information between elements within the computer, such asduring start-up, may be stored in ROM. Further, the computer may includeany device (e.g., computer, laptop, tablet, PDA, cell phone, mobilephone, a smart television, and the like) that is capable of receiving ortransmitting an IP address wirelessly to or from the internet.

The computer may also include a magnetic hard disk drive for readingfrom and writing to a magnetic hard disk, a magnetic disk drive forreading from or writing to a removable magnetic disk, and an opticaldisk drive for reading from or writing to removable optical disk such asa CD-ROM or other optical media. The magnetic hard disk drive, magneticdisk drive, and optical disk drive are connected to the system bus by ahard disk drive interface, a magnetic disk drive-interface, and anoptical drive interface, respectively. The drives and their associatedcomputer-readable media provide nonvolatile storage ofcomputer-executable instructions, data structures, program modules, andother data for the computer. Although the exemplary environmentdescribed herein employs a magnetic hard disk, a removable magneticdisk, and a removable optical disk, other types of computer readablemedia for storing data can be used, including magnetic cassettes, flashmemory cards, digital video disks, Bernoulli cartridges, RAMs, ROMs,SSDs, and the like.

Communication media typically embody computer readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media.

One or more aspects of the disclosure may be embodied incomputer-executable instructions (i.e., software), routines, orfunctions stored in system memory or nonvolatile memory as applicationprograms, program modules, and/or program data. The software mayalternatively be stored remotely, such as on a remote computer withremote application programs. Generally, program modules includeroutines, programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data typeswhen executed by a processor in a computer or other device. The computerexecutable instructions may be stored on one or more tangible,non-transitory computer readable media (e.g., hard disk, optical disk,removable storage media, solid state memory, RAM, etc.) and executed byone or more processors or other devices. As will be appreciated by oneof skill in the art, the functionality of the program modules may becombined or distributed as desired in various embodiments. In addition,the functionality may be embodied in whole or in part in firmware orhardware equivalents such as integrated circuits, application specificintegrated circuits, field programmable gate arrays (FPGA), and thelike.

The computer may operate in a networked environment using logicalconnections to one or more remote computers. The remote computers mayeach be another personal computer, a tablet, a PDA, a server, a router,a network PC, a peer device, or other common network node, and typicallyinclude many or all of the elements described above relative to thecomputer. The logical connections include a local area network (LAN) anda wide area network (WAN) that are presented here by way of example andnot limitation. Such networking environments are commonplace inoffice-wide or enterprise-wide computer networks, intranets and theInternet.

When used in a LAN networking environment, the computer is connected tothe local network through a network interface or adapter. When used in aWAN networking environment, the computer may include a modem, a wirelesslink, or other means for establishing communications over the wide areanetwork, such as the Internet. The modem, which may be internal orexternal, is connected to the system bus via the serial port interface.In a networked environment, program modules depicted relative to thecomputer, or portions thereof, may be stored in the remote memorystorage device. It will be appreciated that the network connectionsshown are exemplary and other means of establishing communications overwide area network may be used.

Preferably, computer-executable instructions are stored in a memory,such as the hard disk drive, and executed by the computer.Advantageously, the computer processor has the capability to perform alloperations (e.g., execute computer-executable instructions) inreal-time.

The order of execution or performance of the operations in embodimentsillustrated and described herein is not essential, unless otherwisespecified. That is, the operations may be performed in any order, unlessotherwise specified, and embodiments may include additional or feweroperations than those disclosed herein. For example, it is contemplatedthat executing or performing a particular operation before,contemporaneously with, or after another operation is within the scopeof aspects of the disclosure.

Embodiments may be implemented with computer-executable instructions.The computer-executable instructions may be organized into one or morecomputer-executable components or modules. Aspects of the disclosure maybe implemented with any number and organization of such components ormodules. For example, aspects of the disclosure are not limited to thespecific computer-executable instructions or the specific components ormodules illustrated in the figures and described herein. Otherembodiments may include different computer-executable instructions orcomponents having more or less functionality than illustrated anddescribed herein.

When introducing elements of aspects of the disclosure or theembodiments thereof, the articles “a”, “an”, “the” and “said” areintended to mean that there are one or more of the elements. The terms“comprising”, “including”, and “having” are intended to be inclusive andmean that there may be additional elements other than the listedelements.

Having described aspects of the disclosure in detail, it will beapparent that modifications and variations are possible withoutdeparting from the scope of aspects of the disclosure as defined in theappended claims. As various changes could be made in the aboveconstructions, products, and methods without departing from the scope ofaspects of the disclosure, it is intended that all matter contained inthe above description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

What is claimed is:
 1. A monitoring system, comprising: at least onemonitor device configured to monitor one or more physical properties ofa user; an artificial intelligence (AI) system configured to receive andanalyze the monitored physical properties to predict when the user is ina deep sleep cycle state of sleep and to generate one or more activityparameters optimized or personalized to the user, wherein said one ormore activity parameters corresponds to a patient room order to avoiddisturbing the user when the AI system predicts the user is in the deepsleep cycle state of sleep; and a smart alert system, the smart alertsystem in communication with the monitor device and configured tomonitor and record physical properties of the user during a time periodleading up to an activation of an alert by the user, the smart alertsystem configured to proactively send out a signal for help before theuser activates the alert based on a future activation predicted by theAI system.
 2. The monitoring system of claim 1, wherein the AI systemand monitor device are in wireless communication with one another. 3.The monitoring system of claim 2, further comprising a portablecomputing device in wireless communication with the monitor device andAI system, the portable computing device configured to receive the oneor more physical properties of the user and send the one or morephysical properties of the user to the AI system, wherein the portablecomputing device is configured to receive the one or more activityparameters from the AI system and present the one or more activityparameters to the user.
 4. The monitoring system of claim 3, wherein onemonitor device is a biometric sensor configured to be implanted in theuser.
 5. The monitoring system of claim 2, wherein the AI system isconfigured to store the monitored physical properties.
 6. The monitoringsystem of claim 5, wherein the AI system is further configured toanalyze the monitored physical properties to predict a biologic functionof the user and/or determine a user's response to the one or moreactivity parameters.
 7. The monitoring system of claim 1, wherein the AIsystem implements one or more of predictive learning, machine learning,automated planning and scheduling, machine perception, computer visionand affective computing to generate said one or more activity parametersoptimized or personalized to the user.
 8. The monitoring system of claim1, wherein the smart alert system is configured to compile the one ormore physical properties of the user every time the user presses a callbutton to create a set of conditions for activation of the alert.
 9. Themonitoring system of claim 8, wherein the smart alert system sends outthe signal for help when the one or more physical properties are withina predetermined threshold of the set of conditions.
 10. The monitoringsystem of claim 1, wherein the patient room order comprises a medicationschedule, the AI system configured to access the medication schedule andto send a signal to administer medicine based on the medication scheduleand the state of sleep of the user.
 11. The monitoring system of claim1, wherein the patient room order comprises an order for a bloodpressure measurement, the AI system configured to activate the bloodpressure measurement based on the state of sleep of the user.
 12. Amonitoring system comprising at least one external, non-implantedmonitor device configured to monitor one or more physical properties ofa user, wherein the one or more physical properties of the usercomprises at least one of the following: calories consumption, state ofsleep, quality of sleep, activity, heart rhythm, pulse rate, bloodpressure, temperature, perspiration, and blood chemistry; a smart alertsystem associated with a call button, the smart alert system incommunication with the monitor device and configured to monitor andrecord physical properties of the user during a time period leading upto an activation of the call button by the user; and an artificialintelligence (AI) system configured to receive and analyze the monitoredand recorded physical properties to predict a future activation of thecall button by the user, the smart alert system configured toproactively send out a signal for help before the user activates thecall button based on the future activation predicted by the AI system.13. The monitoring system of claim 12, wherein the smart alert systemsends out the signal for help when the one or more physical propertiesare within a predetermined threshold of the set of conditions.